gemini
gemini copied to clipboard
scylladb
An Insert query fails with: "Mutation failed: mutation error: gocql: connection closed waiting for response"
Argus: https://argus.scylladb.com/tests/scylla-cluster-tests/3357a196-9963-49d6-850a-0afb4bfbdf4c
gemini-gocql-driver v1.16.1 2025-10-26T16:28:51Z e1344c7234e3576e9c6c7fd250f69b0016a5468e gemini 2.2.3 2025-12-29T21:47:13Z 0f450d840199e72ed8676a5b8c2287006f41b894
SCT event:
2026-01-04 12:11:30.072: (TestFrameworkEvent Severity.ERROR) period_type=one-time event_id=e9a3edfe-f3d8-4f2a-b119-b1c5b2336278, source=GeminiTest.test_load_random_with_nemesis()
exception=self = <gemini_test.GeminiTest testMethod=test_load_random_with_nemesis>
def test_load_random_with_nemesis(self):
cmd = self.params.get("gemini_cmd")
self.db_cluster.add_nemesis(nemesis=self.get_nemesis_class(), tester_obj=self)
self.log.debug("Start gemini benchmark")
gemini_thread = self.run_gemini(cmd=cmd)
self.gemini_results["cmd"] = gemini_thread.gemini_commands
# sleep before run nemesis test_duration * .25
# sleep_before_start = float(self.params.get('test_duration')) * 60 * .1
sleep_before_start = 120
self.log.info('Sleep interval {}'.format(sleep_before_start))
time.sleep(sleep_before_start)
self.db_cluster.start_nemesis()
# self.run_post_prepare_cql_cmds()
self.gemini_results.update(self.verify_gemini_results(queue=gemini_thread))
self.db_cluster.stop_nemesis(timeout=1600)
> self.verify_results()
gemini_test.py:62:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
gemini_test.py:120: in verify_results
self.fail(self.gemini_results["results"])
E AssertionError: [{'errors': [{'timestamp': '2026-01-04T11:44:44.957399351Z', 'err': {'start_time': '2026-01-04T11:44:44.79736144Z', 'end_time': '2026-01-04T11:44:44.957377003Z', 'final_error': 'mutation error: gocql: connection closed waiting for response (potentially executed: false), partition keys: {"pk0":[0.5335549939939157],"pk1":["e281d261-9adb-4038-b8d5-1f0769876761"]}', 'statement': {'PartitionKeys': {'Values': {'pk0': [0.5335549939939157], 'pk1': ['e281d261-9adb-4038-b8d5-1f0769876761']}}, 'Query': 'INSERT INTO ks1.table1 (pk0,pk1,ck0,ck1,col0,col1,col2,col3,col4,col5,col6,col7,col8,col9) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?) ', 'Values': [0.5335549939939157, 'e281d261-9adb-4038-b8d5-1f0769876761', 77960970325122, '5b2dee77bbdd6e371b0d3e9fcfe7f379bcde49a4522994ca65b244221188c4e2f', {'udt_1777407658_0': 0.41081464, 'udt_1777407658_1': True, 'udt_1777407658_2': '7393-11-06T09:40:06.999Z', 'udt_1777407658_3': '9810-10-26T01:53:51.798Z'}, 6795521149337159415, 0.9797844757440872, 95303316928220, 'ac6e3f178bcdeeff79e4fa7d369b45a2dd369b4da', 0.04755962, ['a0d864321100804140a050281c06030c7e371381c0e8743d7ebf5fafd7e3f9fb1d8ecf67b3d9ecfb158ac56abd56ab56bb55a2d964ba5d2c864', 'ae31188c46a3512814bedfef773b9d4e272c964ba55229148ad369b4daed763b9d55aa55aa55aa55aa552a95cae572399', 'c6bb5daedf6fb7dbe0e87c3e170b85c2e13090', '8dc6e34b2592c9643299ccf8fcfeff7f3f1f0fff7f3f9f4f279349a8d46a359a4d26937ebfdfef773b9dce22'], '189.79.136.92', '43.149.176.240', 4809597230333788461], 'QueryType': 'InsertStatement'}, 'operation': 'mutation', 'total_attempts': 11}, 'partition-keys': {'pk0': [0.5335549939939157], 'pk1': ['e281d261-9adb-4038-b8d5-1f0769876761']}, 'message': 'Mutation failed: mutation error: gocql: connection closed waiting for response (potentially executed: false), partition keys: {"pk0":[0.5335549939939157],"pk1":["e281d261-9adb-4038-b8d5-1f0769876761"]}\n\nStore status: test=false, oracle=true', 'query': 'INSERT INTO ks1.table1 (pk0,pk1,ck0,ck1,col0,col1,col2,col3,col4,col5,col6,col7,col8,col9) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?) ', 'values': [0.5335549939939157, 'e281d261-9adb-4038-b8d5-1f0769876761', 77960970325122, '5b2dee77bbdd6e371b0d3e9fcfe7f379bcde49a4522994ca65b244221188c4e2f', {'udt_1777407658_0': 0.41081464, 'udt_1777407658_1': True, 'udt_1777407658_2': '7393-11-06T09:40:06.999Z', 'udt_1777407658_3': '9810-10-26T01:53:51.798Z'}, 6795521149337159415, 0.9797844757440872, 95303316928220, 'ac6e3f178bcdeeff79e4fa7d369b45a2dd369b4da', 0.04755962, ['a0d864321100804140a050281c06030c7e371381c0e8743d7ebf5fafd7e3f9fb1d8ecf67b3d9ecfb158ac56abd56ab56bb55a2d964ba5d2c864', 'ae31188c46a3512814bedfef773b9d4e272c964ba55229148ad369b4daed763b9d55aa55aa55aa55aa552a95cae572399', 'c6bb5daedf6fb7dbe0e87c3e170b85c2e13090', '8dc6e34b2592c9643299ccf8fcfeff7f3f1f0fff7f3f9f4f279349a8d46a359a4d26937ebfdfef773b9dce22'], '189.79.136.92', '43.149.176.240', 4809597230333788461], 'stmt-type': 'InsertStatement'}], 'write_ops': 68687732, 'write_errors': 32, 'read_ops': 67271298, 'validated_rows': 29495728, 'read_errors': 0}]
The gemini result log had:
{
"gemini_version": "2.2.3",
"result": {
"errors": [
{
"timestamp": "2026-01-04T11:44:44.957399351Z",
"err": {
"start_time": "2026-01-04T11:44:44.79736144Z",
"end_time": "2026-01-04T11:44:44.957377003Z",
"final_error": "mutation error: gocql: connection closed waiting for response (potentially executed: false), partition keys: {\"pk0\":[0.5335549939939157],\"pk1\":[\"e281d261-9adb-4038-b8d5-1f0769876761\"]}",
"statement": {
"PartitionKeys": {
"Values": {
"pk0": [
0.5335549939939157
],
"pk1": [
"e281d261-9adb-4038-b8d5-1f0769876761"
]
}
},
"Query": "INSERT INTO ks1.table1 (pk0,pk1,ck0,ck1,col0,col1,col2,col3,col4,col5,col6,col7,col8,col9) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?) ",
"Values": [
...
@yarongilor Please share more details about what was happening at the cluster at that time. If you feel like there was nothing important please state what you checked as well
Not fully sure about the scenario and testing branch. have to retest it.
Closing this issue as it was moved to Jira. Please continue the thread in https://scylladb.atlassian.net/browse/QATOOLS-118
