scylladb
A gemini "row count differ" event is too long and unclear
https://argus.scylladb.com/tests/scylla-cluster-tests/1b37479f-c06d-484b-8c81-f78700eaf448 event:
2025-12-31 19:57:15.239: (TestFrameworkEvent Severity.ERROR) period_type=one-time event_id=d28d1353-105c-46cc-a2c2-891969ec3c70, source=GeminiTest.test_load_random_with_nemesis()
exception=self = <gemini_test.GeminiTest testMethod=test_load_random_with_nemesis>
def test_load_random_with_nemesis(self):
cmd = self.params.get("gemini_cmd")
self.db_cluster.add_nemesis(nemesis=self.get_nemesis_class(), tester_obj=self)
self.log.debug("Start gemini benchmark")
gemini_thread = self.run_gemini(cmd=cmd)
self.gemini_results["cmd"] = gemini_thread.gemini_commands
# sleep before run nemesis test_duration * .25
sleep_before_start = float(self.params.get("test_duration")) * 60 * 0.1
self.log.info("Sleep interval {}".format(sleep_before_start))
time.sleep(sleep_before_start)
self.db_cluster.start_nemesis()
self.gemini_results.update(self.verify_gemini_results(queue=gemini_thread))
self.db_cluster.stop_nemesis(timeout=1600)
> self.verify_results()
gemini_test.py:61:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
gemini_test.py:119: in verify_results
self.fail(self.gemini_results["results"])
E AssertionError: [{'errors': [{'timestamp': '2025-12-31T17:04:53.436991463Z', 'err': {'start_time': '2025-12-31T17:04:44.240531865Z', 'end_time': '2025-12-31T17:04:53.436985629Z', 'final_error': 'pk: pk0=5155-06-22 04:15:29, pk1=b9bcd6633190c06030000000000000000b95caed7eb75badda0d0e8743a1d8ec7b0d8ec, pk2=426036f0-a562-1781-b052-0a854b9af98f, pk3=0xc0f9b3c500, ck0=9bc5eaf472311088442a1d0ecf6fb7dbe5f2f97e8f47a3d1e0f87436733198cc6e371b8fc7ebfdfef773b9d74ba5d2e97cbe, ck1=ec1fd610-ad20-1100-a7c4-0a854b9af98f, ck2=c4ea7d369c, ck3=7bbd5eafd793c9e4f2793c1e0f\n- ck0: 9bc5eaf472311088442a1d0ecf6fb7dbe5f2f97e8f47a3d1e0f87436733198cc6e371b8fc7ebfdfef773b9d74ba5d2e97cbe\n+ ck0: 67b3d9ecf67b3db95caed7eb753a9d9f4f279349a4d26\n- ck1: ec1fd610-ad20-1100-a7c4-0a854b9af98f\n+ ck1: 44577b80-81aa-1ffe-a57c-0a854b9af98f\n- ck2: c4ea7d369c\n+ ck2: 5\n- ck3: 7bbd5eafd793c9e4f2793c1e0f\n+ ck3: 20a85c261b0582c16d369b45aa\n- col0: 5369-07-16 07:55:56\n+ col0: 2833-07-20 11:45:09\n- col1: [8714122164710011259 9219474001605107536 3900380666569004728 3635260063975729379 4012524695421895854 7163703625277054719 5708975509273505682 8605945422730139074]\n+ col1: [443427127699259208 3727265498074511891 2307706077675697819 6392532961248955520 6916087400482280180 3568990978710222408 7267359831702436411]\n- col10: 0.0678800884517401\n+ col10: 0.8792503834114753\n- col2: 0.75844455\n+ col2: 0.39225727\n- col3: 337856221\n+ col3: 1881205812\n- col4: 635221396\n+ col4: 57242738\n- col5: 74.142.149.126\n+ col5: 136.36.32.42\n- col6: 4403541744333424362\n+ col6: 7012155003821516956\n- col7: {0 15740 1322459573725}\n+ col7: {0 33550 57130249007772}\n- col8: 0.964096\n+ col8: 0.2888173\n- col9: 5687336098656218505\n+ col9: 7344757350850229335\n', 'statement': {'PartitionKeys': {'Values': {'pk0': [66620210458915, 35967419430222, 100523736929539], 'pk1': ['6f47abddeef77bb5d7b3d9e4fa7532914edf67b3d1e0f87434', '7ebf5faf57abd5d369341a8d462391140a058241a0d0682592c964b259ac56d5ea753a1d0e87', 'b9bcd6633190c06030000000000000000b95caed7eb75badda0d0e8743a1d8ec7b0d8ec'], 'pk2': ['d5de42d0-8753-1c3c-9bf6-0a854b9af98f', 'ea106850-5e53-1abc-a9c9-0a854b9af98f', '426036f0-a562-1781-b052-0a854b9af98f'], 'pk3': [5385377912401289800, 6372517596052346107, 7328329593060604445]}}, 'Query': 'SELECT pk0,pk1,pk2,pk3,ck0,ck1,ck2,ck3,col0,col1,col2,col3,col4,col5,col6,col7,col8,col9,col10 FROM ks1.table1 WHERE pk0 IN (?,?,?) AND pk1 IN (?,?,?) AND pk2 IN (?,?,?) AND pk3 IN (?,?,?) ', 'Values': [66620210458915, 35967419430222, 100523736929539, '6f47abddeef77bb5d7b3d9e4fa7532914edf67b3d1e0f87434', '7ebf5faf57abd5d369341a8d462391140a058241a0d0682592c964b259ac56d5ea753a1d0e87', 'b9bcd6633190c06030000000000000000b95caed7eb75badda0d0e8743a1d8ec7b0d8ec', 'd5de42d0-8753-1c3c-9bf6-0a854b9af98f', 'ea106850-5e53-1abc-a9c9-0a854b9af98f', '426036f0-a562-1781-b052-0a854b9af98f', 5385377912401289800, 6372517596052346107, 7328329593060604445], 'QueryType': 'SelectMultiPartitionType'}, 'operation': 'validation', 'total_attempts': 10}, 'partition-keys': {'pk0': [66620210458915, 35967419430222, 100523736929539], 'pk1': ['6f47abddeef77bb5d7b3d9e4fa7532914edf67b3d1e0f87434', '7ebf5faf57abd5d369341a8d462391140a058241a0d0682592c964b259ac56d5ea753a1d0e87', 'b9bcd6633190c06030000000000000000b95caed7eb75badda0d0e8743a1d8ec7b0d8ec'], 'pk2': ['d5de42d0-8753-1c3c-9bf6-0a854b9af98f', 'ea106850-5e53-1abc-a9c9-0a854b9af98f', '426036f0-a562-1781-b052-0a854b9af98f'], 'pk3': [5385377912401289800, 6372517596052346107, 7328329593060604445]}, 'message': 'Validation failed: pk: pk0=5155-06-22 04:15:29, pk1=b9bcd6633190c06030000000000000000b95caed7eb75badda0d0e8743a1d8ec7b0d8ec, pk2=426036f0-a562-1781-b052-0a854b9af98f, pk3=0xc0f9b3c500, ck0=9bc5eaf472311088442a1d0ecf6fb7dbe5f2f97e8f47a3d1e0f87436733198cc6e371b8fc7ebfdfef773b9d74ba5d2e97cbe, ck1=ec1fd610-ad20-1100-a7c4-0a854b9af98f, ck2=c4ea7d369c, ck3=7bbd5eafd793c9e4f2793c1e0f\n- ck0: 9bc5eaf472311088442a1d0ecf6fb7dbe5f2f97e8f47a3d1e0f87436733198cc6e371b8fc7ebfdfef773b9d74ba5d2e97cbe\n+ ck0: 67b3d9ecf67b3db95caed7eb753a9d9f4f279349a4d26\n- ck1: ec1fd610-ad20-1100-a7c4-0a854b9af98f\n+ ck1: 44577b80-81aa-1ffe-a57c-0a854b9af98f\n- ck2: c4ea7d369c\n+ ck2: 5\n- ck3: 7bbd5eafd793c9e4f2793c1e0f\n+ ck3: 20a85c261b0582c16d369b45aa\n- col0: 5369-07-16 07:55:56\n+ col0: 2833-07-20 11:45:09\n- col1: [8714122164710011259 9219474001605107536 3900380666569004728 3635260063975729379 4012524695421895854 7163703625277054719 5708975509273505682 8605945422730139074]\n+ col1: [443427127699259208 3727265498074511891 2307706077675697819 6392532961248955520 6916087400482280180 3568990978710222408 7267359831702436411]\n- col10: 0.0678800884517401\n+ col10: 0.8792503834114753\n- col2: 0.75844455\n+ col2: 0.39225727\n- col3: 337856221\n+ col3: 1881205812\n- col4: 635221396\n+ col4: 57242738\n- col5: 74.142.149.126\n+ col5: 136.36.32.42\n- col6: 4403541744333424362\n+ col6: 7012155003821516956\n- col7: {0 15740 1322459573725}\n+ col7: {0 33550 57130249007772}\n- col8: 0.964096\n+ col8: 0.2888173\n- col9: 5687336098656218505\n+ col9: 7344757350850229335\n', 'query': 'SELECT pk0,pk1,pk2,pk3,ck0,ck1,ck2,ck3,col0,col1,col2,col3,col4,col5,col6,col7,col8,col9,col10 FROM ks1.table1 WHERE pk0 IN (?,?,?) AND pk1 IN (?,?,?) AND pk2 IN (?,?,?) AND pk3 IN (?,?,?) ', 'values': [66620210458915, 35967419430222, 100523736929539, '6f47abddeef77bb5d7b3d9e4fa7532914edf67b3d1e0f87434', '7ebf5faf57abd5d369341a8d462391140a058241a0d0682592c964b259ac56d5ea753a1d0e87', 'b9bcd6633190c06030000000000000000b95caed7eb75badda0d0e8743a1d8ec7b0d8ec', 'd5de42d0-8753-1c3c-9bf6-0a854b9af98f', 'ea106850-5e53-1abc-a9c9-0a854b9af98f', '426036f0-a562-1781-b052-0a854b9af98f', 5385377912401289800, 6372517596052346107, 7328329593060604445], 'stmt-type': 'SelectMultiPartitionType'}, {'timestamp': '2025-12-31T17:05:02.286231355Z', 'err': {'start_time': '2025-12-31T17:04:52.073155028Z', 'end_time': '2025-12-31T17:05:02.286228516Z', 'final_error': 'pk: pk0=7082-08-11 23:21:19, pk1=211008e8f4fa7dbe5f2f974da, pk2=c8b58860-f6e1-141e-b2fc-0a854b9af98f, pk3=0xc03c9e3150, ck0=deef2090c86432198cc6763b9dce67b3d96c502894ca6532190c4f279349a4d269b4170b8542a1d068b47fbfdfeff7f, ck1=046e6d50-cd25-1fae-8fe6-0a854b9af98f, ck2=f279349a4d2e9f43e1f0f8743a1d0683a9dce67b3592c16349a4da653a9d, ck3=b1d86c3610088442219048a44aa5d269b45aad56bedfef773b1d8ec7954aa55229140a85188cc6e371b85caed86c361\n- ck0: deef2090c86432198cc6763b9dce67b3d96c502894ca6532190c4f279349a4d269b4170b8542a1d068b47fbfdfeff7f\n+ ck0: ddeef7fb29944aa5d2e9f4fa4ca6d3e9743a9d4e4ba5522994cae5f26e379bcd66b3592c4321108844a25128cae572391c8e\n- ck1: 046e6d50-cd25-1fae-8fe6-0a854b9af98f\n+ ck1: 5e2d79d0-b57f-1a5d-af68-0a854b9af98f\n- ck2: f279349a4d2e9f43e1f0f8743a1d0683a9dce67b3592c16349a4da653a9d\n+ ck2: aa55b359acd6eb753a1dc86432198cc6e3f1e7f3f97cbe5fafd75b2d9\n- ck3: b1d86c3610088442219048a44aa5d269b45aad56bedfef773b1d8ec7954aa55229140a85188cc6e371b85caed86c361\n+ ck3: 28140a85c2619f4f271309040201160b85c2e1f078bc170b85c2e170b85c90c864b2d9ecf67b26130904\n- col0: 9842-09-30 12:47:12\n+ col0: 8299-01-28 08:27:01\n- col1: [6060977453908048379 637967153207481450 6329110924720562887 2692414864933924475]\n+ col1: [2899558715443430665 1265881621503461758 6282622680097164095 6396921439466135666 4709347814225530029 6290159718506801184]\n- col10: 0.3472121290496394\n+ col10: 0.25911144432506283\n- col2: 0.045832753\n+ col2: 0.32940167\n- col3: 2022266191\n+ col3: 1494186868\n- col4: 314047261\n+ col4: 1773473441\n- col5: 144.135.114.239\n+ col5: 112.99.166.134\n- col6: 1996269409418602556\n+ col6: 1615657674670359218\n- col7: {0 11129 6945947204322}\n+ col7: {0 66337 5134673389299}\n- col8: 0.7387198\n+ col8: 0.9686605\n- col9: 5219153300016260788\n+ col9: 7718173098126250733\n', 'statement': {'PartitionKeys': {'Values': {'pk0': [141397948984616, 120213081178374, 161338432879232], 'pk1': ['a158a8ac5e2f178bcdeef3a1d', 'afd7e3f1f0f8743218643a1d0e8743a9de472', '211008e8f4fa7dbe5f2f974da'], 'pk2': ['be6d3630-da78-1482-a869-0a854b9af98f', '3768fac0-94dd-1e8c-a11b-0a854b9af98f', 'c8b58860-f6e1-141e-b2fc-0a854b9af98f'], 'pk3': [8397034890080057235, 2346807892535029853, 8812406691224325396]}}, 'Query': 'SELECT pk0,pk1,pk2,pk3,ck0,ck1,ck2,ck3,col0,col1,col2,col3,col4,col5,col6,col7,col8,col9,col10 FROM ks1.table1 WHERE pk0 IN (?,?,?) AND pk1 IN (?,?,?) AND pk2 IN (?,?,?) AND pk3 IN (?,?,?) ', 'Values': [141397948984616, 120213081178374, 161338432879232, 'a158a8ac5e2f178bcdeef3a1d', 'afd7e3f1f0f8743218643a1d0e8743a9de472', '211008e8f4fa7dbe5f2f974da', 'be6d3630-da78-1482-a869-0a854b9af98f', '3768fac0-94dd-1e8c-a11b-0a854b9af98f', 'c8b58860-f6e1-141e-b2fc-0a854b9af98f', 8397034890080057235, 2346807892535029853, 8812406691224325396], 'QueryType': 'SelectMultiPartitionType'}, 'operation': 'validation', 'total_attempts': 10}, 'partition-keys': {'pk0': [141397948984616, 120213081178374, 161338432879232], 'pk1': ['a158a8ac5e2f178bcdeef3a1d', 'afd7e3f1f0f8743218643a1d0e8743a9de472', '211008e8f4fa7dbe5f2f974da'], 'pk2': ['be6d3630-da78-1482-a869-0a854b9af98f', '3768fac0-94dd-1e8c-a11b-0a854b9af98f', 'c8b58860-f6e1-141e-b2fc-0a854b9af98f'], 'pk3': [8397034890080057235, 2346807892535029853, 8812406691224325396]}, 'message': 'Validation failed: pk: pk0=7082-08-11 23:21:19, pk1=211008e8f4fa7dbe5f2f974da, pk2=c8b58860-f6e1-141e-b2fc-0a854b9af98f, pk3=0xc03c9e3150, ck0=deef2090c86432198cc6763b9dce67b3d96c502894ca6532190c4f279349a4d269b4170b8542a1d068b47fbfdfeff7f, ck1=046e6d50-cd25-1fae-8fe6-0a854b9af98f, ck2=f279349a4d2e9f43e1f0f8743a1d0683a9dce67b3592c16349a4da653a9d, ck3=b1d86c3610088442219048a44aa5d269b45aad56bedfef773b1d8ec7954aa55229140a85188cc6e371b85caed86c361\n- ck0: deef2090c86432198cc6763b9dce67b3d96c502894ca6532190c4f279349a4d269b4170b8542a1d068b47fbfdfeff7f\n+ ck0: ddeef7fb29944aa5d2e9f4fa4ca6d3e9743a9d4e4ba5522994cae5f26e379bcd66b3592c4321108844a25128cae572391c8e\n- ck1: 046e6d50-cd25-1fae-8fe6-0a854b9af98f\n+ ck1: 5e2d79d0-b57f-1a5d-af68-0a854b9af98f\n- ck2: f279349a4d2e9f43e1f0f8743a1d0683a9dce67b3592c16349a4da653a9d\n+ ck2: aa55b359acd6eb753a1dc86432198cc6e3f1e7f3f97cbe5fafd75b2d9\n- ck3: b1d86c3610088442219048a44aa5d269b45aad56bedfef773b1d8ec7954aa55229140a85188cc6e371b85caed86c361\n+ ck3: 28140a85c2619f4f271309040201160b85c2e1f078bc170b85c2e170b85c90c864b2d9ecf67b26130904\n- col0: 9842-09-30 12:47:12\n+ col0: 8299-01-28 08:27:01\n- col1: [6060977453908048379 637967153207481450 6329110924720562887 2692414864933924475]\n+ col1: [2899558715443430665 1265881621503461758 6282622680097164095 6396921439466135666 4709347814225530029 6290159718506801184]\n- col10: 0.3472121290496394\n+ col10: 0.25911144432506283\n- col2: 0.045832753\n+ col2: 0.32940167\n- col3: 2022266191\n+ col3: 1494186868\n- col4: 314047261\n+ col4: 1773473441\n- col5: 144.135.114.239\n+ col5: 112.99.166.134\n- col6: 1996269409418602556\n+ col6: 1615657674670359218\n- col7: {0 11129 6945947204322}\n+ col7: {0 66337 5134673389299}\n- col8: 0.7387198\n+ col8: 0.9686605\n- col9: 5219153300016260788\n+ col9: 7718173098126250733\n', 'query': 'SELECT pk0,pk1,pk2,pk3,ck0,ck1,ck2,ck3,col0,col1,col2,col3,col4,col5,col6,col7,col8,col9,col10 FROM ks1.table1 WHERE pk0 IN (?,?,?) AND pk1 IN (?,?,?) AND pk2 IN (?,?,?) AND pk3 IN (?,?,?) ', 'values': [141397948984616, 120213081178374, 161338432879232, 'a158a8ac5e2f178bcdeef3a1d', 'afd7e3f1f0f8743218643a1d0e8743a9de472', '211008e8f4fa7dbe5f2f974da', 'be6d3630-da78-1482-a869-0a854b9af98f', '3768fac0-94dd-1e8c-a11b-0a854b9af98f', 'c8b58860-f6e1-141e-b2fc-0a854b9af98f', 8397034890080057235, 2346807892535029853, 8812406691224325396], 'stmt-type': 'SelectMultiPartitionType'}, {'timestamp': '2025-12-31T17:06:40.228631994Z', 'err': {'start_time': '2025-12-31T17:06:30.058218669Z', 'end_time': '2025-12-31T17:06:40.228628823Z', 'final_error': 'pk: pk0=6758-07-25 06:34:02, pk1=2c96cb6532198c078, pk2=1b24ca30-ff34-1b67-ae76-0a854b9af98f, pk3=0xc0c4c0de88, ck0=feff7fb7db5da6db65b2d168bc96432994c261389562b150a8542, ck1=3851fb40-7766-1f07-9c5b-0a854b9af98f, ck2=482412090482412f9fc7ebf5faf57ab3f9fcf67b3d96cb6da6d369b4da6d369150a85c2613098cc90c86432994ca653eaf5fafd7e, ck3=100804020180974ba552a9d46a3581c0e07038\n- ck0: feff7fb7db5da6db65b2d168bc96432994c261389562b150a8542\n+ ck0: fc7ebf5fafd76b4aa5d269341a0d869acd66b3592c168b29148a4522914824ebf57abd5e2f974b0a\n- ck1: 3851fb40-7766-1f07-9c5b-0a854b9af98f\n+ ck1: fa5b99d0-3b69-12ce-ad50-0a854b9af98f\n- ck2: 482412090482412f9fc7ebf5faf57ab3f9fcf67b3d96cb6da6d369b4da6d369150a85c2613098cc90c86432994ca653eaf5fafd7e\n+ ck2: 379bcde6fb7ce67b359ac56abd57c3e1f0f87c361b0190c86c361b058ac2793c9643219\n- ck3: 100804020180974ba552a9d46a3581c0e07038\n+ ck3: b2d96c369b4da552a9d4ea753a9de472b95caed7ebf5a653a9542a95ca65542a150a8542a1d0d96cb6dbed76bb5d1b0d068341a0d0e8fd7e3f9f4fa7532\n- col0: 3854-03-23 00:55:52\n+ col0: 5121-10-04 19:48:11\n- col1: [2824085020895739676 1674640086966492298 6135095734630531736 5587511730999905056]\n+ col1: [5090475450262671451 3016843724276886490 8104219274140522709 5684185187022040815 810505188003953601]\n- col10: 0.4709201509020474\n+ col10: 0.46882470398440956\n- col2: 0.7665629\n+ col2: 0.61161274\n- col3: 229519005\n+ col3: 1010548419\n- col4: 999139585\n+ col4: 778087402\n- col5: 126.8.242.210\n+ col5: 51.107.89.41\n- col6: 9018258163026457329\n+ col6: 7812940071179924248\n- col7: {0 82650 48384794000991}\n+ col7: {0 17144 77415519965889}\n- col8: 0.25509888\n+ col8: 0.22724968\n- col9: 1435837612855585484\n+ col9: 662168959411101924\n', 'statement': {'PartitionKeys': {'Values': {'pk0': [151112414042907, 61667320934265], 'pk1': ['2c96cb6532198c078', '389bddeeff7fbfdfe7fd369349a4d2613095d2e97cbe5f279bc6231984c261389c4ab55aa55aa55'], 'pk2': ['1b24ca30-ff34-1b67-ae76-0a854b9af98f', 'db52b140-324e-12c1-9810-0a854b9af98f'], 'pk3': [8827333440560156514, 8764256609106829482]}}, 'Query': 'SELECT pk0,pk1,pk2,pk3,ck0,ck1,ck2,ck3,col0,col1,col2,col3,col4,col5,col6,col7,col8,col9,col10 FROM ks1.table1 WHERE pk0 IN (?,?) AND pk1 IN (?,?) AND pk2 IN (?,?) AND pk3 IN (?,?) ', 'Values': [151112414042907, 61667320934265, '2c96cb6532198c078', '389bddeeff7fbfdfe7fd369349a4d2613095d2e97cbe5f279bc6231984c261389c4ab55aa55aa55', '1b24ca30-ff34-1b67-ae76-0a854b9af98f', 'db52b140-324e-12c1-9810-0a854b9af98f', 8827333440560156514, 8764256609106829482], 'QueryType': 'SelectMultiPartitionType'}, 'operation': 'validation', 'total_attempts': 10}, 'partition-keys': {'pk0': [151112414042907, 61667320934265], 'pk1': ['2c96cb6532198c078', '389bddeeff7fbfdfe7fd369349a4d2613095d2e97cbe5f279bc6231984c261389c4ab55aa55aa55'], 'pk2': ['1b24ca30-ff34-1b67-ae76-0a854b9af98f', 'db52b140-324e-12c1-9810-0a854b9af98f'], 'pk3': [8827333440560156514, 8764256609106829482]}, 'message': 'Validation failed: pk: pk0=6758-07-25 06:34:02, pk1=2c96cb6532198c078, pk2=1b24ca30-ff34-1b67-ae76-0a854b9af98f, pk3=0xc0c4c0de88, ck0=feff7fb7db5da6db65b2d168bc96432994c261389562b150a8542, ck1=3851fb40-7766-1f07-9c5b-0a854b9af98f, ck2=482412090482412f9fc7ebf5faf57ab3f9fcf67b3d96cb6da6d369b4da6d369150a85c2613098cc90c86432994ca653eaf5fafd7e, ck3=100804020180974ba552a9d46a3581c0e07038\n- ck0: feff7fb7db5da6db65b2d168bc96432994c261389562b150a8542\n+ ck0: fc7ebf5fafd76b4aa5d269341a0d869acd66b3592c168b29148a4522914824ebf57abd5e2f974b0a\n- ck1: 3851fb40-7766-1f07-9c5b-0a854b9af98f\n+ ck1: fa5b99d0-3b69-12ce-ad50-0a854b9af98f\n- ck2: 482412090482412f9fc7ebf5faf57ab3f9fcf67b3d96cb6da6d369b4da6d369150a85c2613098cc90c86432994ca653eaf5fafd7e\n+ ck2: 379bcde6fb7ce67b359ac56abd57c3e1f0f87c361b0190c86c361b058ac2793c9643219\n- ck3: 100804020180974ba552a9d46a3581c0e07038\n+ ck3: b2d96c369b4da552a9d4ea753a9de472b95caed7ebf5a653a9542a95ca65542a150a8542a1d0d96cb6dbed76bb5d1b0d068341a0d0e8fd7e3f9f4fa7532\n- col0: 3854-03-23 00:55:52\n+ col0: 5121-10-04 19:48:11\n- col1: [2824085020895739676 1674640086966492298 6135095734630531736 5587511730999905056]\n+ col1: [5090475450262671451 3016843724276886490 8104219274140522709 5684185187022040815 810505188003953601]\n- col10: 0.4709201509020474\n+ col10: 0.46882470398440956\n- col2: 0.7665629\n+ col2: 0.61161274\n- col3: 229519005\n+ col3: 1010548419\n- col4: 999139585\n+ col4: 778087402\n- col5: 126.8.242.210\n+ col5: 51.107.89.41\n- col6: 9018258163026457329\n+ col6: 7812940071179924248\n- col7: {0 82650 48384794000991}\n+ col7: {0 17144 77415519965889}\n- col8: 0.25509888\n+ col8: 0.22724968\n- col9: 1435837612855585484\n+ col9: 662168959411101924\n', 'query': 'SELECT pk0,pk1,pk2,pk3,ck0,ck1,ck2,ck3,col0,col1,col2,col3,col4,col5,col6,col7,col8,col9,col10 FROM ks1.table1 WHERE pk0 IN (?,?) AND pk1 IN (?,?) AND pk2 IN (?,?) AND pk3 IN (?,?) ', 'values': [151112414042907, 61667320934265, '2c96cb6532198c078', '389bddeeff7fbfdfe7fd369349a4d2613095d2e97cbe5f279bc6231984c261389c4ab55aa55aa55', '1b24ca30-ff34-1b67-ae76-0a854b9af98f', 'db52b140-324e-12c1-9810-0a854b9af98f', 8827333440560156514, 8764256609106829482], 'stmt-type': 'SelectMultiPartitionType'}, {'timestamp': '2025-12-31T17:06:46.811931079Z', 'err': {'start_time': '2025-12-31T17:06:37.109167787Z', 'end_time': '2025-12-31T17:06:46.811928273Z', 'final_error': 'pk: pk0=7636-11-12 19:11:35, pk1=188c462b1d86c0301008040a0d0e8fd7ebf5fafd76bb5ca6532190c0683, pk2=1fa8ba00-6839-11ab-8578-0a854b9af98f, pk3=0xc058329f70, ck0=e9743ad5eaf5fafdfe7fbf91c864, ck1=f053d1c0-3f2f-1ce0-a1b9-0a854b9af98f, ck2=753a8d46a351a0d86435aad56abd5eaf57a9fcf67b359acd66bfeff7f3f1f0f87c348a4d269349, ck3=ee7f3f9fc9acde673b9dcee77269349a4d2e9f47a0884422110884422e170b85caed7eb754b25128944a25\n- ck0: e9743ad5eaf5fafdfe7fbf91c864\n+ ck0: e8f4723914\n- ck1: f053d1c0-3f2f-1ce0-a1b9-0a854b9af98f\n+ ck1: 4b23b720-55dd-12e3-84ce-0a854b9af98f\n- ck2: 753a8d46a351a0d86435aad56abd5eaf57a9fcf67b359acd66bfeff7f3f1f0f87c348a4d269349\n+ ck2: 391c0e87c361b08c4623118844a2d1b2592c160b8542214d26130984c2e170cfe7f3\n- ck3: ee7f3f9fc9acde673b9dcee77269349a4d2e9f47a0884422110884422e170b85caed7eb754b25128944a25\n+ ck3: ada3512894cae5f2f973b95cae57abd5ea170b85c2e1\n- col0: 6992-10-25 06:28:21\n+ col0: 7406-02-15 10:54:20\n- col1:
...
...
@yarongilor event is long, because you set error limit to 1000 and this is not just one error but 40 collected during this run. Indeed it's not easy to follow, but I took one of it and put into LLM with this investigation result:
This looks like you’re not actually comparing “the same row” — you’re comparing different clustering rows from the same partition key.
Why I’m confident:
Your query filters only the partition key components (pk0..pk3) using IN (...) and does not constrain the clustering keys (ck0..ck3).
The diff shows all ck* fields differ (ck0..ck3), and then all non-key columns differ too. That’s exactly what happens when each DB returns a different row within the same partition.
In Cassandra/Scylla-like models:
Rows within a single partition are ordered by clustering keys, but
When you query multiple partitions (IN on partition key), the overall row order across partitions is not guaranteed, and even within a partition you can still end up comparing “first row” vs “first row” if your validator does row-by-row comparison without sorting/aligning by full primary key.
Common validator failure mode:
DB A returns row with (pk..., ck=9bc5..., ec1f..., c4ea...)
DB B returns row with (pk..., ck=67b3..., 4457..., 5, ...)
Validator compares them as if they correspond → mismatch.
How to fix it (any one of these works):
Align rows by full primary key before comparing Sort/group results by (pk0,pk1,pk2,pk3, ck0,ck1,ck2,ck3) in the validator, then compare.
Make the query uniquely identify rows
Add clustering predicates (e.g., AND ck0=? AND ck1=? AND ck2=? AND ck3=?) if you intend single-row checks.
Add an ORDER BY (only helps within a partition)
ORDER BY ck0, ck1, ck2, ck3 won’t fully solve cross-partition ordering, but can reduce surprises if you compare per-partition.
Also double-check your validator isn’t doing something like “compare first row only per partition” or “compare nth row”, because with multi-partition reads that’s fragile.
If you paste how the validator matches rows (by position vs by key), I can point to the exact line of logic that makes this fail.
@CodeLieutenant This could be gemini issue but @yarongilor this could be also Scylladb issue that is shown when test cluster missed a row (or had too many rows). @CodeLieutenant validator should first make sure we have the same ck's for each partition before validating other fields (there's no point in validating them if we compare different rows). This way error message should show: row with ck's... was missing in test/oracle
We need to fix this error message.
@soyacz you are completly right, we are sorting based only on primary keys, and not on clustering, this is not a difficult change, I'll try to do it as soon as possible
@yarongilor event is long, because you set error limit to 1000 and this is not just one error but 40 collected during this run. Indeed it's not easy to follow, but I took one of it and put into LLM with this investigation result:
This looks like you’re not actually comparing “the same row” — you’re comparing different clustering rows from the same partition key.
Why I’m confident:
Your query filters only the partition key components (pk0..pk3) using IN (...) and does not constrain the clustering keys (ck0..ck3).
The diff shows all ck* fields differ (ck0..ck3), and then all non-key columns differ too. That’s exactly what happens when each DB returns a different row within the same partition.
In Cassandra/Scylla-like models:
Rows within a single partition are ordered by clustering keys, but
When you query multiple partitions (IN on partition key), the overall row order across partitions is not guaranteed, and even within a partition you can still end up comparing “first row” vs “first row” if your validator does row-by-row comparison without sorting/aligning by full primary key.
Common validator failure mode:
DB A returns row with (pk..., ck=9bc5..., ec1f..., c4ea...)
DB B returns row with (pk..., ck=67b3..., 4457..., 5, ...)
Validator compares them as if they correspond → mismatch.
How to fix it (any one of these works):
Align rows by full primary key before comparing Sort/group results by (pk0,pk1,pk2,pk3, ck0,ck1,ck2,ck3) in the validator, then compare.
Make the query uniquely identify rows
Add clustering predicates (e.g., AND ck0=? AND ck1=? AND ck2=? AND ck3=?) if you intend single-row checks.
Add an ORDER BY (only helps within a partition)
ORDER BY ck0, ck1, ck2, ck3 won’t fully solve cross-partition ordering, but can reduce surprises if you compare per-partition.
Also double-check your validator isn’t doing something like “compare first row only per partition” or “compare nth row”, because with multi-partition reads that’s fragile.
If you paste how the validator matches rows (by position vs by key), I can point to the exact line of logic that makes this fail.
@CodeLieutenant This could be gemini issue but @yarongilor this could be also Scylladb issue that is shown when test cluster missed a row (or had too many rows). @CodeLieutenant validator should first make sure we have the same ck's for each partition before validating other fields (there's no point in validating them if we compare different rows). This way error message should show: row with ck's... was missing in test/oracle
We need to fix this error message.
As for the number of errors - SCT set it to 1000 by default. This doesn't make sense so i sent a fix to lower it in https://github.com/scylladb/scylla-cluster-tests/pull/13101
Closing this issue as it was moved to Jira. Please continue the thread in https://scylladb.atlassian.net/browse/QATOOLS-117
