[sui-node] thread 'main' panicked at 'no entry found for key'

[velvia]

Steps to Reproduce Issue

Sometimes when full node is started and the disk has not been wiped there is this unhelpful error message:

2022-11-02 01:56:10 | thread 'main' panicked at 'no entry found for key', /sui/crates/sui-core/src/authority_aggregator.rs:742:31
-- | --
  |   | 2022-11-02 01:56:10 | 2022-11-02T08:56:10.072934Z ERROR telemetry_subscribers: panicked at 'no entry found for key', /sui/crates/sui-core/src/authority_aggregator.rs:742:31 panic.file="/sui/crates/sui-core/src/authority_aggregator.rs" panic.line=742 panic.column=31

The full stack trace is like:

2022-11-02 01:56:10 | at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/park/thread.rs:267:31
-- | --
  |   | 2022-11-02 01:56:10 | 34: tokio::park::thread::CachedParkThread::block_on
  |   | 2022-11-02 01:56:10 | at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:72:5
  |   | 2022-11-02 01:56:10 | 33: tokio::coop::budget
  |   | 2022-11-02 01:56:10 | at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:95:5
  |   | 2022-11-02 01:56:10 | 32: tokio::coop::with_budget
  |   | 2022-11-02 01:56:10 | at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/thread/local.rs:421:9
  |   | 2022-11-02 01:56:10 | 31: std::thread::local::LocalKey<T>::with
  |   | 2022-11-02 01:56:10 | at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/thread/local.rs:445:16
  |   | 2022-11-02 01:56:10 | 30: std::thread::local::LocalKey<T>::try_with
  |   | 2022-11-02 01:56:10 | at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:102:9
  |   | 2022-11-02 01:56:10 | 29: tokio::coop::with_budget::{{closure}}
  |   | 2022-11-02 01:56:10 | at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/park/thread.rs:267:54
  |   | 2022-11-02 01:56:10 | 28: tokio::park::thread::CachedParkThread::block_on::{{closure}}
  |   | 2022-11-02 01:56:10 | at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tracing-0.1.37/src/instrument.rs:272:9
  |   | 2022-11-02 01:56:10 | 27: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
  |   | 2022-11-02 01:56:10 | at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/future/mod.rs:91:19
  |   | 2022-11-02 01:56:10 | 26: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  |   | 2022-11-02 01:56:10 | at ./crates/sui-node/src/main.rs:70:70
  |   | 2022-11-02 01:56:10 | 25: sui_node::main::{{closure}}
  |   | 2022-11-02 01:56:10 | at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/future/mod.rs:91:19
  |   | 2022-11-02 01:56:10 | 24: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  |   | 2022-11-02 01:56:10 | at ./crates/sui-node/src/lib.rs:240:83
  |   | 2022-11-02 01:56:10 | 23: sui_node::SuiNode::start::{{closure}}
  |   | 2022-11-02 01:56:10 | at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/future/mod.rs:91:19
  |   | 2022-11-02 01:56:10 | 22: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  |   | 2022-11-02 01:56:10 | at ./crates/sui-core/src/authority_active.rs:264:71
  |   | 2022-11-02 01:56:10 | 21: sui_core::authority_active::ActiveAuthority<A>::sync_to_latest_checkpoint::{{closure}}
  |   | 2022-11-02 01:56:10 | at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/future/mod.rs:91:19
  |   | 2022-11-02 01:56:10 | 20: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  |   | 2022-11-02 01:56:10 | at ./crates/sui-core/src/authority_active.rs:281:10
  |   | 2022-11-02 01:56:10 | 19: sui_core::authority_active::ActiveAuthority<A>::sync_to_latest_checkpoint_with_config::{{closure}}
  |   | 2022-11-02 01:56:10 | at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/future/mod.rs:91:19
  |   | 2022-11-02 01:56:10 | 18: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  |   | 2022-11-02 01:56:10 | at ./crates/sui-core/src/authority_active/checkpoint_driver/mod.rs:513:10
  |   | 2022-11-02 01:56:10 | 17: sui_core::authority_active::checkpoint_driver::get_latest_checkpoint_from_all::{{closure}}
  |   | 2022-11-02 01:56:10 | at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/future/mod.rs:91:19
  |   | 2022-11-02 01:56:10 | 16: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  |   | 2022-11-02 01:56:10 | at ./crates/sui-core/src/authority_aggregator.rs:715:10
  |   | 2022-11-02 01:56:10 | 15: sui_core::authority_aggregator::AuthorityAggregator<A>::quorum_map_then_reduce_with_timeout::{{closure}}
  |   | 2022-11-02 01:56:10 | at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/future/mod.rs:91:19
  |   | 2022-11-02 01:56:10 | 14: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  |   | 2022-11-02 01:56:10 | at ./crates/sui-core/src/authority_aggregator.rs:739:67
  |   | 2022-11-02 01:56:10 | 13: sui_core::authority_aggregator::AuthorityAggregator<A>::quorum_map_then_reduce_with_timeout_and_prefs::{{closure}}
  |   | 2022-11-02 01:56:10 | at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/iter/traits/iterator.rs:1836:9
  |   | 2022-11-02 01:56:10 | 12: core::iter::traits::iterator::Iterator::collect
  |   | 2022-11-02 01:56:10 | at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.24/src/stream/futures_unordered/mod.rs:635:9
  |   | 2022-11-02 01:56:10 | 11: <futures_util::stream::futures_unordered::FuturesUnordered<Fut> as core::iter::traits::collect::FromIterator<Fut>>::from_iter
  |   | 2022-11-02 01:56:10 | at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/iter/adapters/map.rs:124:9
  |   | 2022-11-02 01:56:10 | 10: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::fold
  |   | 2022-11-02 01:56:10 | at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/iter/traits/iterator.rs:2414:21
  |   | 2022-11-02 01:56:10 | 9: core::iter::traits::iterator::Iterator::fold
  |   | 2022-11-02 01:56:10 | at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/iter/adapters/map.rs:84:28
  |   | 2022-11-02 01:56:10 | 8: core::iter::adapters::map::map_fold::{{closure}}
  |   | 2022-11-02 01:56:10 | at ./crates/sui-core/src/authority_aggregator.rs:742:31
  |   | 2022-11-02 01:56:10 | 7: sui_core::authority_aggregator::AuthorityAggregator<A>::quorum_map_then_reduce_with_timeout_and_prefs::{{closure}}::{{closure}}
  |   | 2022-11-02 01:56:10 | at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/alloc/src/collections/btree/map.rs:2219:9
  |   | 2022-11-02 01:56:10 | 6: <alloc::collections::btree::map::BTreeMap<K,V,A> as core::ops::index::Index<&Q>>::index
  |   | 2022-11-02 01:56:10 | at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/option.rs:738:21
  |   | 2022-11-02 01:56:10 | 5: core::option::Option<T>::expect
  |   | 2022-11-02 01:56:10 | at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/option.rs:1874:5
  |   | 2022-11-02 01:56:10 | 4: core::option::expect_failed
  |   | 2022-11-02 01:56:10 | at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/panicking.rs:56:5
  |   | 2022-11-02 01:56:10 | 3: core::panicking::panic_str
  |   | 2022-11-02 01:56:10 | at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/panicking.rs:72:5
  |   | 2022-11-02 01:56:10 | 2: core::panicking::panic_display
  |   | 2022-11-02 01:56:10 | at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/panicking.rs:142:14
  |   | 2022-11-02 01:56:10 | 1: core::panicking::panic_fmt
  |   | 2022-11-02 01:56:10 | at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panicking.rs:584:5
  |   | 2022-11-02 01:56:10 | 0: rust_begin_unwind
  |   | 2022-11-02 01:56:10 | stack backtrace:
  |   | 2022-11-02 01:56:10 | thread 'main' panicked at 'no entry found for key', /sui/crates/sui-core/src/authority_aggregator.rs:742:31
  |   | 2022-11-02 01:56:10 | 2022-11-02T08:56:10.072934Z ERROR telemetry_subscribers: panicked at 'no entry found for key', /sui/crates/sui-core/src/authority_aggregator.rs:742:31 panic.file="/sui/crates/sui-core/src/authority_aggregator.rs" panic.line=742 panic.column=31

The place in the code where this happens is:

        // First, execute in parallel for each authority FMap.
        let mut responses: futures::stream::FuturesUnordered<_> = authorities_shuffled
            .iter()
            .map(|name| {
                let client = &self.authority_clients[name];

Basically the "name" is missing from authority_clients.

Expected Result

We should at the minimum give a more helpful error message.

Context

This was seen when 0.13.3 was deployed to alternative full nodes and the disk was not wiped. Wiping the disk solved the issue and this was not seen anymore.