Enabling the `elasticsearch` feature makes Zebra panic

[upbqdn]

Starting Zebra with the elasticsearch feature enabled, for example, by

cargo run --release --features elasticsearch

causes the following panic:

The application panicked (crashed).
Message:  ES Request should never fail: Error { kind: Http(reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(9200), path: "/zcash_mainnet/_bulk", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })) }) }
Location: zebra-state/src/service/finalized_state.rs:507

Metadata:
version: 1.6.0
Zcash network: Mainnet
running state version: 25.3.0
initial disk state version: 25.3.0
features: default,elasticsearch,getblocktemplate_rpcs,howudoin,indicatif,progress_bar,release_max_level_info
branch: main
git commit: 58bfe97
commit timestamp: 2024-02-23T23:28:44.000000000Z
target triple: x86_64-unknown-linux-gnu
rust compiler: 1.76.0
rust release date: 2024-02-04
optimization level: 3
debug checks: false

...
[1]    8829 IOT instruction (core dumped)  cargo r --features elasticsearch --release

As Arya pointed out, the panic is likely due to an unavailable Kibana server. However, this causes further problems because, for example, running Zebra with --all-features implicitly panics as well.

[oxarbitrage]

I think this is not that bad. If you are running Zebra with the elasticsearch feature enabled and there is no elasticsearch database running the node should stop. I think we might want to replace the panic with an error and proper shutdown.

[upbqdn]

we might want to replace the panic with an error and proper shutdown.

The only problem I see is that we still won't be able to run Zebra with --all-features without the server, but that shouldn't be too much of an issue.

[oxarbitrage]

Another option can be something like:

• check for the database server if the feature is enabled, log an error and keep a flag if needed.
• if the feature is enabled but the flag is false ignore elasticsearch code.

[upbqdn]

I guess we deal with something similar when we enable metrics with the prometheus feature, but Prometheus is not available. Zebra can handle that without panicking.

[upbqdn]

Zebra can handle that without panicking.

Or stopping. I just checked that, and it just continues running.