Sui local fullnode Bug

[FrankC01]

trafficstars

Steps to Reproduce Issue

Setup in home dir:

1. mkdir sui_local
2. sui genesis --working-dir sui_local
3. `sui start --network.config sui_local/network.yaml'
4. `sui-node --config-path sui_local/fullnode.yaml'

This worked first time: sui client --client.config sui_local/client.yaml gas

But sui client --client.config sui_local/client.yaml objects

Fails as does all subsequent sui client... calls

Expected Result

A list of objects owned by the address

Actual Result

thread 'main' panicked at 'Sui System State object must always exist', crates/sui-core/src/authority/authority_store_tables.rs:218:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Abort trap: 6

And:

$ RUST_BACKTRACE=1 sui client --client.config sui_local/client.yaml objects
thread 'main' panicked at 'Sui System State object must always exist', crates/sui-core/src/authority/authority_store_tables.rs:218:14
stack backtrace:
   0: _rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::panicking::panic_display
   3: core::panicking::panic_str
   4: core::option::expect_failed
   5: sui_core::authority::authority_store_tables::AuthorityPerpetualTables<S>::get_sui_system_state_object
   6: sui_core::authority::authority_store_tables::AuthorityPerpetualTables<S>::get_epoch
   7: sui_core::authority::authority_store::SuiDataStore<S>::open
   8: sui_core::gateway_state::GatewayState<sui_core::authority_client::NetworkAuthorityClient>::create_client
   9: sui_sdk::SuiClient::new_embedded_client
  10: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  11: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  12: sui::sui_commands::SuiCommand::execute::{{closure}}
  13: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  14: tokio::park::thread::CachedParkThread::block_on
  15: tokio::runtime::Runtime::block_on
  16: sui::main

System Information

• OS: macos Monterey 12.6.1 M1 Max

[gnomadic]

I am able to reproduce this:

OS: macos Monterey 12.5.1 M1 Max ==> rust: stable 1.64.0 (bottled), HEAD cargo 1.64.0 (387270bc7 2022-09-16) sui: version = "0.13.2" (from https://github.com/MystenLabs/sui/blob/devnet/Cargo.toml)

terminal 1:

➜  ~ sui genesis -f 
...
➜  ~ sui start

terminal 2:

➜  sui sui client addresses
Showing 5 results.
0x0c7494bf22fab998838b953e4a5a402157a52b97
0x2d4ede8c9cee98b7e4ee1a2b5dd9a7d463fc9178
0x4975c04133637b4a5a7f79086e3fad71a9fa8a37
0x92ce114fa14c6d6bf2ec3ec2e1493a02b85f5d46
0xf2ca6d34cdd92abf251ec4cb7be9de54bd406f59
➜  sui bash -c "RUST_BACKTRACE=full sui client addresses"
thread 'main' panicked at 'Sui System State object must always exist', crates/sui-core/src/authority/authority_store_tables.rs:218:14
stack backtrace:
   0:        0x10b0e2774 - std::backtrace_rs::backtrace::libunwind::trace::h0934ce8016da62ea
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:        0x10b0e2774 - std::backtrace_rs::backtrace::trace_unsynchronized::h59c3dee870344ec2
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:        0x10b0e2774 - std::sys_common::backtrace::_print_fmt::h498b4033e357b134
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/sys_common/backtrace.rs:66:5
   3:        0x10b0e2774 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h7d80e14dd19335c9
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/sys_common/backtrace.rs:45:22
   4:        0x10b1050db - core::fmt::write::h1709d0255080e28a
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/fmt/mod.rs:1198:17
   5:        0x10b0da778 - std::io::Write::write_fmt::hecc56b5639d319e7
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/io/mod.rs:1672:15
   6:        0x10b0e42fd - std::sys_common::backtrace::_print::h6f0d53db9de204c5
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/sys_common/backtrace.rs:48:5
   7:        0x10b0e42fd - std::sys_common::backtrace::print::h33928b4e9b284b1e
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/sys_common/backtrace.rs:35:9
   8:        0x10b0e42fd - std::panicking::default_hook::{{closure}}::had4d1cd22a173020
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panicking.rs:295:22
   9:        0x10b0e404e - std::panicking::default_hook::h42aa124509888735
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panicking.rs:314:9
  10:        0x1091df238 - telemetry_subscribers::set_panic_hook::{{closure}}::h024dfb35691d0a4c
  11:        0x10b0e48fb - std::panicking::rust_panic_with_hook::h2b231e816574a23a
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panicking.rs:702:17
  12:        0x10b0e4773 - std::panicking::begin_panic_handler::{{closure}}::h9da8d88b7a4c9d5e
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panicking.rs:588:13
  13:        0x10b0e2bf7 - std::sys_common::backtrace::__rust_end_short_backtrace::h10dbf1377dfaf877
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/sys_common/backtrace.rs:138:18
  14:        0x10b0e444a - rust_begin_unwind
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panicking.rs:584:5
  15:        0x10b22b293 - core::panicking::panic_fmt::hde1544b10dc8b4d3
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/panicking.rs:142:14
  16:        0x10b10263b - core::panicking::panic_display::h5ea834a07ce22ae1
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/panicking.rs:72:5
  17:        0x10b1025ec - core::panicking::panic_str::h0bec6afd6497d506
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/panicking.rs:56:5
  18:        0x10b22b129 - core::option::expect_failed::h88974b902a3f0e9b
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/option.rs:1874:5
  19:        0x108cb5db1 - sui_core::authority::authority_store_tables::AuthorityPerpetualTables<S>::get_sui_system_state_object::h1e93eac1537b1b1c
  20:        0x108cb85b2 - sui_core::authority::authority_store::SuiDataStore<S>::open::he907fe82dce22ea0
  21:        0x108d7c546 - sui_core::gateway_state::GatewayState<sui_core::authority_client::NetworkAuthorityClient>::create_client::ha263aa9630f1ecb2
  22:        0x10892836d - sui_sdk::SuiClient::new_embedded_client::he46120fc8774a6c2
  23:        0x1082acfe6 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::hf8cb5abd73cfbeab
  24:        0x10829ab9d - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h71c3e7b59302ff33
  25:        0x10825c5b6 - sui::sui_commands::SuiCommand::execute::{{closure}}::h727faf73b7043250
  26:        0x1082a3f90 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::hb1624a1d2f22833c
  27:        0x1082dc55b - tokio::park::thread::CachedParkThread::block_on::hc4c2b09ec6866cee
  28:        0x1083927f5 - tokio::runtime::Runtime::block_on::h1b93555fe19622c5
  29:        0x108363306 - sui::main::h0af7ca6eeb547402
  30:        0x1082580d6 - std::sys_common::backtrace::__rust_begin_short_backtrace::h104815e89ad788d3
  31:        0x108307a7c - std::rt::lang_start::{{closure}}::h3d5edb4a01f03319
  32:        0x10b0d23bb - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::ha6b934dd2f7d5008
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/ops/function.rs:280:13
  33:        0x10b0d23bb - std::panicking::try::do_call::h5a3f4b34abdbdae8
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panicking.rs:492:40
  34:        0x10b0d23bb - std::panicking::try::h56b1848524fc332f
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panicking.rs:456:19
  35:        0x10b0d23bb - std::panic::catch_unwind::h44afff85dd6d04a7
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panic.rs:137:14
  36:        0x10b0d23bb - std::rt::lang_start_internal::{{closure}}::hd7c11497b2582199
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/rt.rs:128:48
  37:        0x10b0d23bb - std::panicking::try::do_call::h2353d56408689f1d
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panicking.rs:492:40
  38:        0x10b0d23bb - std::panicking::try::h58a401d784e0ca5d
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panicking.rs:456:19
  39:        0x10b0d23bb - std::panic::catch_unwind::h35fff46a669b8467
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panic.rs:137:14
  40:        0x10b0d23bb - std::rt::lang_start_internal::h18b571735d69908e
                               at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/rt.rs:128:20
  41:        0x1083635a9 - _main
[1]    46830 abort      bash -c "RUST_BACKTRACE=full sui client addresses"

[patrickkuo]

this issue is cause by this https://github.com/MystenLabs/sui/blob/da34fcc6e36ea5702258c55ba5f444cb545c9c85/crates/sui-core/src/authority/authority_store.rs#L84-L88

This is only affecting “embedded gateway” client, it is expecting the 0x5 object to be available if the store is not new, however for gateway state we don’t pull objects automatically, so 0x5 won’t be in the store (if you query the 0x5 object sui client object --id 0x5 first, subsequent sui client call will not fail ).

The assumption of 0x5 object is always stored in a old data store is probably incorrect, the fullnode/validator will fail the same way if it didn’t record the 0x5 object into the database when it first started.

For now we can use PRC instead sui client switch --rpc http://127.0.0.1:9000/

For short term fix we can load the 0x5 object when the Sui CLI starts, for long term fix we should review the get_epoch logic or remove support of "embedded gateway" connection mode.

[gnomadic]

where is port 9000 specified? sui client switch --rpc http://127.0.0.1:9000/ fails with HTTP error: error trying to connect: tcp connect error: Connection refused (os error 61)

When I run lsof -Pn -i4 | grep "sui" I don't see port 9000 open. I also don't see it ~/.sui/sui_config/networks.yaml

Any way I can confirm what port the RPC service is running on?

[patrickkuo]

where is port 9000 specified? sui client switch --rpc http://127.0.0.1:9000/ fails with HTTP error: error trying to connect: tcp connect error: Connection refused (os error 61)

When I run lsof -Pn -i4 | grep "sui" I don't see port 9000 open. I also don't see it ~/.sui/sui_config/networks.yaml

Any way I can confirm what port the RPC service is running on?

You will need to run sui-node --config-path ~/.sui/sui_config/fullnode.yaml to start the fullnode, which host the RPC service

[patrickkuo]

this is fixed by https://github.com/MystenLabs/sui/pull/5751, will be available after next devnet update

[FrankC01]

Looks promising, and I like the short cuts for changing environments by name versus setting IPs!

[awelc]

Since, per @patrickkuo's comment above, the fix was provided in #5751, I am closing this issue.