sui icon indicating copy to clipboard operation
sui copied to clipboard

Published 20 hours ago • verified publisher icon MystenLabs

Data store corruption when running Sui with the unsecure signature scheme

Open jonas-lj opened this issue 2 years ago • 0 comments

Steps to Reproduce Issue

cargo run --package sui-benchmark --bin stress -- --disjoint-mode true --num-client-threads 3 --num-server-threads 12 --num-transfer-accounts 2 bench --target-qps 5000 --in-flight-ratio 3 --num-workers 10 --run-duration 30s`.

Note that this fork uses a trivial, unsecure signature scheme for validators.

Expected Result

I would expect the stress test to run without issues, and with a very high tps.

Actual Result

I am getting a lot of errors.

jonaslindstrom@jonaslindstrom sui-benchmark % cargo run --package sui-benchmark --bin stress -- --disjoint-mode true --num-client-threads 3 --num-server-threads 12 --num-transfer-accounts 2 bench --target-qps 5000 --in-flight-ratio 3 --num-workers 10 --run-duration 30s
    Finished dev [unoptimized + debuginfo] target(s) in 1.27s
     Running `/Users/jonaslindstrom/Documents/fork/sui/target/debug/stress --disjoint-mode true --num-client-threads 3 --num-server-threads 12 --num-transfer-accounts 2 bench --target-qps 5000 --in-flight-ratio 3 --num-workers 10 --run-duration 30s`
Configuring local benchmark..
Creating enough gas to transfer objects..
2022-10-31T08:33:52.696155Z ERROR narwhal_consensus::bullshark: The store does not contain the parent of mxye7UxivBiXOtH6: C56(yVhAaGKzugvIg1a7NEykcvrp9/T8BJ0xoS+PY7OVBmoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA, OvUpiDuYW8rNhhJe, E0): Missing item digest=KP0AYXnx4nhNKzVAXuCaIM1zv0Wt6Uj4lYG6oRhL5Ts=
2022-10-31T08:34:20.783576Z ERROR anemo_tower::trace::on_failure: response failed error=Error: sending stopped by peer: error 0 latency=1 ms
Creating objects to transfer..
Creating enough gas to transfer objects..
2022-10-31T08:35:15.852266Z ERROR narwhal_consensus::bullshark: The store does not contain the parent of sKEkYSOHavBYoMsF: C614(j/7tGLiTSi90urhj3NbIf2uYn7T/sIOmASNOax/TVMEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA, 88QEcLsWNOxQaiiD, E0): Missing item digest=AO+ElsXRluHDBU/tC3yptKBeI87749GpXIpvG+zZ3Z0=
2022-10-31T08:35:15.852419Z ERROR narwhal_consensus::bullshark: The store does not contain the parent of sKEkYSOHavBYoMsF: C614(j/7tGLiTSi90urhj3NbIf2uYn7T/sIOmASNOax/TVMEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA, 88QEcLsWNOxQaiiD, E0): Missing item digest=AO+ElsXRluHDBU/tC3yptKBeI87749GpXIpvG+zZ3Z0=
Creating objects to transfer..
Creating enough gas to transfer objects..
Creating objects to transfer..
Creating enough gas to transfer objects..
2022-10-31T08:36:45.395737Z ERROR narwhal_consensus::bullshark: The store does not contain the parent of yTiJuwQLGV2Qjm6C: C1218(S+NND8odeWMa5nKFwRvzmgWXuZiKoMsP1Ne7akkzOOMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA, jRbPENf4IVKFmIbl, E0): Missing item digest=F2NJnH4Xjfn9+aVryP1kpoxniLOaZOzGYEjHMMpvxkw=
2022-10-31T08:36:50.639565Z ERROR narwhal_consensus::bullshark: The store does not contain the parent of VxpN47iqQxx5IX9H: C1250(S+NND8odeWMa5nKFwRvzmgWXuZiKoMsP1Ne7akkzOOMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA, ZfuWMdAgPjU6smd2, E0): Missing item digest=cZcNaRwxjEvx58DfuH3trSK6GZ6IIzGzZklWFClIEug=
2022-10-31T08:36:50.641135Z ERROR narwhal_consensus::bullshark: The store does not contain the parent of VxpN47iqQxx5IX9H: C1250(S+NND8odeWMa5nKFwRvzmgWXuZiKoMsP1Ne7akkzOOMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA, ZfuWMdAgPjU6smd2, E0): Missing item digest=cZcNaRwxjEvx58DfuH3trSK6GZ6IIzGzZklWFClIEug=
2022-10-31T08:36:50.641792Z ERROR narwhal_consensus::bullshark: The store does not contain the parent of VxpN47iqQxx5IX9H: C1250(S+NND8odeWMa5nKFwRvzmgWXuZiKoMsP1Ne7akkzOOMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA, ZfuWMdAgPjU6smd2, E0): Missing item digest=cZcNaRwxjEvx58DfuH3trSK6GZ6IIzGzZklWFClIEug=
2022-10-31T08:36:52.967648Z ERROR narwhal_consensus::bullshark: The store does not contain the parent of AZhV+9MWTKyGw0mX: C1264(BZc6D699NCkpvbaVsOeMEHazipEwKp+vh40Wq7zH6IUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA, q9rrsUPFNZI3jazs, E0): Missing item digest=FulQg2FprAOAxU6nGlYY/uAQJncWP6ljrdNParVCZRk=
2022-10-31T08:37:05.933293Z ERROR sui_storage::lock_service: Locks did not exist for unsequenced transaction! possible data store corruption tx_digest=CD+9AG4LMp6krhgwHrLTF1sAxemwmSjRXXwOwtWPxN8=
2022-10-31T08:37:05.933450Z ERROR sui_core::authority: commit_certificate failed: Object (0x8cc10918d52423070b26ffc3441db73c32a17e92, SequenceNumber(15893), o#ee1f555dc811ecc3d89c253b4e7f9bfe53eb92b0bd01b136fa1fa9e3d755db5b) lock has not been initialized. digest=CD+9AG4LMp6krhgwHrLTF1sAxemwmSjRXXwOwtWPxN8=
2022-10-31T08:37:05.933512Z ERROR sui_storage::write_ahead_log: DBTxGuard dropped without explicit commit digest=CD+9AG4LMp6krhgwHrLTF1sAxemwmSjRXXwOwtWPxN8=
2022-10-31T08:37:05.938921Z ERROR sui_storage::lock_service: Locks did not exist for unsequenced transaction! possible data store corruption tx_digest=VURgChPu8srkyL68ZPdjEFZ9HNDZnztC++B2uScOwsU=
2022-10-31T08:37:05.939145Z ERROR sui_core::authority: commit_certificate failed: Object (0x8cc10918d52423070b26ffc3441db73c32a17e92, SequenceNumber(15894), o#ab3747ee7f4776c7ca5fef2acbb7122f63a1e56f7c3786abb1cdbf711509b462) lock has not been initialized. digest=VURgChPu8srkyL68ZPdjEFZ9HNDZnztC++B2uScOwsU=
2022-10-31T08:37:05.939253Z ERROR sui_storage::write_ahead_log: DBTxGuard dropped without explicit commit digest=VURgChPu8srkyL68ZPdjEFZ9HNDZnztC++B2uScOwsU=
2022-10-31T08:37:05.942103Z ERROR sui_storage::lock_service: Locks did not exist for unsequenced transaction! possible data store corruption tx_digest=0NT6mngF6vbc05vd1OxwqKkAvTCnnQi1c5yJn9aEpQM=
2022-10-31T08:37:05.942239Z ERROR sui_core::authority: commit_certificate failed: Object (0x8cc10918d52423070b26ffc3441db73c32a17e92, SequenceNumber(15895), o#3f2ee2a0573c2c8d38aef93b0118fd63291868a2238deffb12cf02b7a71ad0cb) lock has not been initialized. digest=0NT6mngF6vbc05vd1OxwqKkAvTCnnQi1c5yJn9aEpQM=
2022-10-31T08:37:05.942284Z ERROR sui_storage::write_ahead_log: DBTxGuard dropped without explicit commit digest=0NT6mngF6vbc05vd1OxwqKkAvTCnnQi1c5yJn9aEpQM=
2022-10-31T08:37:05.942320Z ERROR sui_core::node_sync::node_state: process_digest failed: Object (0x8cc10918d52423070b26ffc3441db73c32a17e92, SequenceNumber(15895), o#3f2ee2a0573c2c8d38aef93b0118fd63291868a2238deffb12cf02b7a71ad0cb) lock has not been initialized. tx_digest=0NT6mngF6vbc05vd1OxwqKkAvTCnnQi1c5yJn9aEpQM=
2022-10-31T08:37:05.942381Z ERROR sui_core::node_sync::node_state: process_digest failed: ExecutionDriver error for 9B7TITULoMhTC/rYfNlG8XALOiO8F0z0yjElOQP3jp4=: Could not execute all parent certificates - Caused by : [ Object (0x8cc10918d52423070b26ffc3441db73c32a17e92, SequenceNumber(15895), o#3f2ee2a0573c2c8d38aef93b0118fd63291868a2238deffb12cf02b7a71ad0cb) lock has not been initialized. ] tx_digest=9B7TITULoMhTC/rYfNlG8XALOiO8F0z0yjElOQP3jp4=
2022-10-31T08:37:05.944458Z ERROR sui_storage::lock_service: Locks did not exist for unsequenced transaction! possible data store corruption tx_digest=0NT6mngF6vbc05vd1OxwqKkAvTCnnQi1c5yJn9aEpQM=
2022-10-31T08:37:05.944625Z ERROR sui_core::authority: commit_certificate failed: Object (0x8cc10918d52423070b26ffc3441db73c32a17e92, SequenceNumber(15895), o#3f2ee2a0573c2c8d38aef93b0118fd63291868a2238deffb12cf02b7a71ad0cb) lock has not been initialized. digest=0NT6mngF6vbc05vd1OxwqKkAvTCnnQi1c5yJn9aEpQM=
2022-10-31T08:37:05.944666Z ERROR sui_storage::write_ahead_log: DBTxGuard dropped without explicit commit digest=0NT6mngF6vbc05vd1OxwqKkAvTCnnQi1c5yJn9aEpQM=
2022-10-31T08:37:05.944694Z ERROR sui_core::node_sync::node_state: process_digest failed: Object (0x8cc10918d52423070b26ffc3441db73c32a17e92, SequenceNumber(15895), o#3f2ee2a0573c2c8d38aef93b0118fd63291868a2238deffb12cf02b7a71ad0cb) lock has not been initialized. tx_digest=0NT6mngF6vbc05vd1OxwqKkAvTCnnQi1c5yJn9aEpQM=
2022-10-31T08:37:05.944757Z ERROR sui_core::node_sync::node_state: process_digest failed: ExecutionDriver error for 9B7TITULoMhTC/rYfNlG8XALOiO8F0z0yjElOQP3jp4=: Could not execute all parent certificates - Caused by : [ Object (0x8cc10918d52423070b26ffc3441db73c32a17e92, SequenceNumber(15895), o#3f2ee2a0573c2c8d38aef93b0118fd63291868a2238deffb12cf02b7a71ad0cb) lock has not been initialized. ] tx_digest=9B7TITULoMhTC/rYfNlG8XALOiO8F0z0yjElOQP3jp4=
2022-10-31T08:37:05.944827Z ERROR sui_core::node_sync::node_state: process_digest failed: ExecutionDriver error for gZO3EDyRQQSfkM9vdcsyTQ0g0Ose+8C5QvjXaGltBeU=: Could not execute all parent certificates - Caused by : [ ExecutionDriver error for 9B7TITULoMhTC/rYfNlG8XALOiO8F0z0yjElOQP3jp4=: Could not execute all parent certificates - Caused by : [ Object (0x8cc10918d52423070b26ffc3441db73c32a17e92, SequenceNumber(15895), o#3f2ee2a0573c2c8d38aef93b0118fd63291868a2238deffb12cf02b7a71ad0cb) lock has not been initialized. ] ] tx_digest=gZO3EDyRQQSfkM9vdcsyTQ0g0Ose+8C5QvjXaGltBeU=
2022-10-31T08:37:05.944899Z ERROR sui_core::node_sync::node_state: process_digest failed: ExecutionDriver error for YsurtLQOUsVuRMDZ0eY924vA97DUekj4ARk/nGRHgJM=: Could not execute all parent certificates - Caused by : [ ExecutionDriver error for gZO3EDyRQQSfkM9vdcsyTQ0g0Ose+8C5QvjXaGltBeU=: Could not execute all parent certificates - Caused by : [ ExecutionDriver error for 9B7TITULoMhTC/rYfNlG8XALOiO8F0z0yjElOQP3jp4=: Could not execute all parent certificates - Caused by : [ Object (0x8cc10918d52423070b26ffc3441db73c32a17e92, SequenceNumber(15895), o#3f2ee2a0573c2c8d38aef93b0118fd63291868a2238deffb12cf02b7a71ad0cb) lock has not been initialized. ] ] ] tx_digest=YsurtLQOUsVuRMDZ0eY924vA97DUekj4ARk/nGRHgJM=
2022-10-31T08:37:05.944979Z ERROR sui_core::node_sync::node_state: process_digest failed: ExecutionDriver error for lyQMBNpo8Vl9QVHypquDmjDxaXFM3CyL6kiOgY0e4UU=: Could not execute all parent certificates - Caused by : [ ExecutionDriver error for YsurtLQOUsVuRMDZ0eY924vA97DUekj4ARk/nGRHgJM=: Could not execute all parent certificates - Caused by : [ ExecutionDriver error for gZO3EDyRQQSfkM9vdcsyTQ0g0Ose+8C5QvjXaGltBeU=: Could not execute all parent certificates - Caused by : [ ExecutionDriver error for 9B7TITULoMhTC/rYfNlG8XALOiO8F0z0yjElOQP3jp4=: Could not execute all parent certificates - Caused by : [ Object (0x8cc10918d52423070b26ffc3441db73c32a17e92, SequenceNumber(15895), o#3f2ee2a0573c2c8d38aef93b0118fd63291868a2238deffb12cf02b7a71ad0cb) lock has not been initialized. ] ] ] ] tx_digest=lyQMBNpo8Vl9QVHypquDmjDxaXFM3CyL6kiOgY0e4UU=
2022-10-31T08:37:05.945770Z ERROR sui_storage::lock_service: Locks did not exist for unsequenced transaction! possible data store corruption tx_digest=9B7TITULoMhTC/rYfNlG8XALOiO8F0z0yjElOQP3jp4=
2022-10-31T08:37:05.946249Z ERROR sui_core::authority: commit_certificate failed: Object (0x8cc10918d52423070b26ffc3441db73c32a17e92, SequenceNumber(15896), o#a094fd5935a89ab6deef9de73bfb4bd03115433629690c48068d3ba1d509af34) lock has not been initialized. digest=9B7TITULoMhTC/rYfNlG8XALOiO8F0z0yjElOQP3jp4=
2022-10-31T08:37:05.946376Z ERROR sui_storage::write_ahead_log: DBTxGuard dropped without explicit commit digest=9B7TITULoMhTC/rYfNlG8XALOiO8F0z0yjElOQP3jp4=
2022-10-31T08:37:05.946986Z ERROR sui_storage::lock_service: Locks did not exist for unsequenced transaction! possible data store corruption tx_digest=gZO3EDyRQQSfkM9vdcsyTQ0g0Ose+8C5QvjXaGltBeU=
2022-10-31T08:37:05.947165Z ERROR sui_core::authority: commit_certificate failed: Object (0x8cc10918d52423070b26ffc3441db73c32a17e92, SequenceNumber(15897), o#34158c29b43cdd36dfccac4e40ac156c8d09a58fad20af67b592db53e03fd840) lock has not been initialized. digest=gZO3EDyRQQSfkM9vdcsyTQ0g0Ose+8C5QvjXaGltBeU=
2022-10-31T08:37:05.947210Z ERROR sui_storage::write_ahead_log: DBTxGuard dropped without explicit commit digest=gZO3EDyRQQSfkM9vdcsyTQ0g0Ose+8C5QvjXaGltBeU=
2022-10-31T08:37:05.947241Z ERROR sui_core::node_sync::node_state: process_digest failed: Object (0x8cc10918d52423070b26ffc3441db73c32a17e92, SequenceNumber(15897), o#34158c29b43cdd36dfccac4e40ac156c8d09a58fad20af67b592db53e03fd840) lock has not been initialized. tx_digest=gZO3EDyRQQSfkM9vdcsyTQ0g0Ose+8C5QvjXaGltBeU=
2022-10-31T08:37:05.947308Z ERROR sui_core::node_sync::node_state: process_digest failed: ExecutionDriver error for YsurtLQOUsVuRMDZ0eY924vA97DUekj4ARk/nGRHgJM=: Could not execute all parent certificates - Caused by : [ Object (0x8cc10918d52423070b26ffc3441db73c32a17e92, SequenceNumber(15897), o#34158c29b43cdd36dfccac4e40ac156c8d09a58fad20af67b592db53e03fd840) lock has not been initialized. ] tx_digest=YsurtLQOUsVuRMDZ0eY924vA97DUekj4ARk/nGRHgJM=
2022-10-31T08:37:05.947378Z ERROR sui_core::node_sync::node_state: process_digest failed: ExecutionDriver error for lyQMBNpo8Vl9QVHypquDmjDxaXFM3CyL6kiOgY0e4UU=: Could not execute all parent certificates - Caused by : [ ExecutionDriver error for YsurtLQOUsVuRMDZ0eY924vA97DUekj4ARk/nGRHgJM=: Could not execute all parent certificates - Caused by : [ Object (0x8cc10918d52423070b26ffc3441db73c32a17e92, SequenceNumber(15897), o#34158c29b43cdd36dfccac4e40ac156c8d09a58fad20af67b592db53e03fd840) lock has not been initialized. ] ] tx_digest=lyQMBNpo8Vl9QVHypquDmjDxaXFM3CyL6kiOgY0e4UU...

The Locks did not exist for unsequenced transaction! possible data store corruption error indicates that rocksdb is behaving incorrectly. Running with debug logging makes it harder to reproduce the errors, indicating that it is related to the higher throughput.

System Information

  • OS: OSX 12.5.1
  • Compiler: rustc 1.65.0 (897e37553 2022-11-02)

jonas-lj avatar Nov 07 '22 21:11 jonas-lj