electrs icon indicating copy to clipboard operation
electrs copied to clipboard
verified publisher icon romanz

Bug: Repeated Database compactification failures

Open leilerg opened this issue 4 months ago • 4 comments

Describe the bug Compacitifcation of index DB fails.

Electrs version v0.10.10

To Reproduce Steps to reproduce the behavior:

  1. Build
  2. Configure and start electrs
  3. Wait
  4. Repeated compactification failures

Expected behavior Initial sync/indexing/compactification to complete without issues.

Configuration

electrs.toml 
Oct 24 16:23:19 btc-apps electrs[3343]: Starting electrs 0.10.10 on x86_64 linux with Config 
{ network: Bitcoin,
  db_path: "/home/electrs/.electrs/index_db/bitcoin", 
  db_log_dir: None,
  db_parallelism: 2, 
  daemon_auth: UserPass("electrs", "<sensitive>"), 
  daemon_rpc_addr: 10.1.1.10:8332, 
  daemon_p2p_addr: 10.1.1.10:8333, 
  electrum_rpc_addr: 10.1.1.11:50001, 
  monitoring_addr: 127.0.0.1:4224, 
  wait_duration: 10s, 
  jsonrpc_timeout: 15s,
  index_batch_size: 10,
  index_lookup_limit: None, 
  reindex_last_blocks: 0,
  auto_reindex: true,
  ignore_mempool: false,
  sync_once: false,
  skip_block_download_wait: false,
  disable_electrum_rpc: false,
  server_banner: "Welcome to electrs 0.10.10 (Electrum Rust Server)!", signet_magic: f9beb4d9 }

Environment variables: Default, unless modified by config Arguments: "--db-parallelism=2" --- See systemd script below

System running electrs

  • Deployment method: Build from source, following instructions in docs Building from source on an Ubuntu 25.04 / Debian 13
  • Debian 13
  • The system is a Proxmox VM with 2 vCPU and 4GB RAM. Host runs on EPYC 7551; VM inherits from host.
  • Bitcoin Core v30 running on another VM with same hw/config; no issues in connectivity encountered; Bitcoin Core running successfully

Electrum client Electrum 4.5.6

Additional context systemd is used to manage the app. The scripts has been adapted from the one in the docs and there's three of them.

The main one, electrs.service:

electrs.service 
electrs@btc-apps:~$ sudo cat /etc/systemd/system/electrs.service 
[Unit]
Description=Electrs Electrum Server
After=bitcoind-online.service


[Service]
WorkingDirectory=/home/electrs/.electrs
ExecStart=/home/electrs/electrs/target/release/electrs --db-parallelism=2
KillMode=process
User=electrs
Group=electrs
Type=simple
Restart=on-failure
TimeoutSec=300
RestartSec=15

Environment="RUST_BACKTRACE=1"


# Hardening measures
PrivateTmp=true
ProtectSystem=full
NoNewPrivileges=true
MemoryDenyWriteExecute=true


[Install]
WantedBy=multi-user.target

bitcoind-online.service is:

bitcoind.service 
electrs@btc-apps:~$ sudo cat /etc/systemd/system/bitcoind-online.service 
[Unit]
Description=Check Bitcoin Core's daemon is online
Requires=network-online.target
After=network-online.target
Before=electrs.service


[Service]
User=electrs
WorkingDirectory=/home/electrs/.electrs
ExecStart=/usr/bin/bash /home/electrs/.electrs/bitcoind-online.sh
Restart=on-failure
TimeoutSec=300
RestartSec=3


[Install]
WantedBy=multi-user.target

Finally, bitcoind-online.sh:

electrs@btc-apps:~$ cat .electrs/bitcoind-online.sh 
#!/usr/bin/bash

curl --user electrs:TheSecretPasswordNoOneCanGuess \
     --data-binary '{"jsonrpc": "2.0", "id": "bitcoind.live", "method": "uptime"}' \
     -H 'content-type: application/json' http://10.1.1.10:8332/

systemd restarts electrs repeatedly and overnight I hit "restart counter is at 65".

Error

systemd logs 
Oct 24 16:23:19 btc-apps electrs[3343]: [2025-10-24T14:23:19.699Z DEBUG tiny_http] Server listening on 127.0.0.1:4224
Oct 24 16:23:19 btc-apps electrs[3343]: [2025-10-24T14:23:19.699Z INFO  electrs::metrics::metrics_impl] serving Prometheus metrics on 127.0.0.1:4224
Oct 24 16:23:19 btc-apps electrs[3343]: [2025-10-24T14:23:19.699Z INFO  electrs::server] serving Electrum RPC on 10.1.1.11:50001
Oct 24 16:23:19 btc-apps electrs[3343]: [2025-10-24T14:23:19.703Z DEBUG tiny_http] Running accept thread
Oct 24 16:23:20 btc-apps electrs[3343]: [2025-10-24T14:23:20.329Z INFO  electrs::db] "/home/electrs/.electrs/index_db/bitcoin": 2897 SST files, 222.522031304 GB, 19.44784542 Grows
Oct 24 16:23:20 btc-apps electrs[3343]: [2025-10-24T14:23:20.329Z DEBUG electrs::db] DB Some(Config { compacted: false, format: 0 })
Oct 24 16:23:21 btc-apps electrs[3343]: [2025-10-24T14:23:21.861Z INFO  electrs::chain] loading 920565 headers, tip=000000000000000000013b0d29be67c2d8118e7217e8cd1f1f675ea4baa0f89b
Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.131Z INFO  electrs::chain] chain updated: tip=000000000000000000013b0d29be67c2d8118e7217e8cd1f1f675ea4baa0f89b, height=920565
Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.133Z DEBUG bitcoincore_rpc] JSON-RPC request: getblockchaininfo []
Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.154Z DEBUG bitcoincore_rpc] JSON-RPC request: getnetworkinfo []
Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.157Z DEBUG bitcoincore_rpc] JSON-RPC request: getnetworkinfo []
Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.158Z DEBUG bitcoincore_rpc] JSON-RPC request: getblockchaininfo []
Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.158Z DEBUG bitcoincore_rpc] JSON-RPC request: getnetworkinfo []
Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.160Z DEBUG electrs::p2p] peer version: VersionMessage { version: 70016, services: ServiceFlags(3081), timestamp: 1761315803, receiver: Address {services: ServiceFlags(NONE), address: 0.0.0.0, port: 0}, sender: Address {services: ServiceFlags(NETWORK|WITNESS|NETWORK_LIMITED|P2P_V2), address: 0.0.0.0, port: 0}, nonce: 16415212804266093349, user_agent: "/Satoshi:30.0.0/", start_height: 920567, relay: true }
Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.160Z DEBUG electrs::p2p] got 2 new headers
Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.160Z INFO  electrs::index] indexing 2 blocks: [920566..920567]
Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.161Z DEBUG electrs::p2p] loading 2 blocks
Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.355Z DEBUG electrs::index] writing 16495 funding and 16685 spending rows from 7134 transactions, 2 blocks
Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.371Z INFO  electrs::chain] chain updated: tip=00000000000000000000b0c20db59ac71bcb893d950bbaf0324063d71b590a5d, height=920567
Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.372Z DEBUG electrs::p2p] got 0 new headers
Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.372Z DEBUG electrs::db] flushing DB column families
Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.539Z INFO  electrs::db] starting config compaction
Oct 24 16:23:23 btc-apps electrs[3343]: [2025-10-24T14:23:23.567Z INFO  electrs::db] starting headers compaction
Oct 24 16:23:26 btc-apps electrs[3343]: [2025-10-24T14:23:26.629Z INFO  electrs::db] starting txid compaction
Oct 24 16:23:29 btc-apps electrs[3343]: [2025-10-24T14:23:29.058Z DEBUG electrs::server] 0: recv {"jsonrpc":"2.0","method":"server.version","id":0,"params":["electrum/4.5.6","1.4"]}
Oct 24 16:33:29 btc-apps electrs[3343]: [2025-10-24T14:33:29.058Z DEBUG electrs::server] 0: disconnected
Oct 24 16:33:29 btc-apps electrs[3343]: [2025-10-24T14:33:29.107Z DEBUG electrs::server] 1: recv {"jsonrpc":"2.0","method":"server.version","id":0,"params":["electrum/4.5.6","1.4"]}
Oct 24 16:35:18 btc-apps electrs[3343]: [2025-10-24T14:35:18.945Z INFO  electrs::db] starting funding compaction
Oct 24 16:35:27 btc-apps electrs[3343]: [2025-10-24T14:35:27.620Z INFO  electrs::db] starting spending compaction
Oct 24 16:35:27 btc-apps electrs[3343]: thread 'main' panicked at src/db.rs:373:14:
Oct 24 16:35:27 btc-apps electrs[3343]: DB::put failed: Error { message: "IO error: While open a file for random read: /home/electrs/.electrs/index_db/bitcoin/001109.sst: Too many open files" }
Oct 24 16:35:27 btc-apps electrs[3343]: stack backtrace:
Oct 24 16:35:27 btc-apps electrs[3343]:    0: rust_begin_unwind
Oct 24 16:35:27 btc-apps electrs[3343]:    1: core::panicking::panic_fmt
Oct 24 16:35:27 btc-apps electrs[3343]:    2: core::result::unwrap_failed
Oct 24 16:35:27 btc-apps electrs[3343]:    3: electrs::db::DBStore::set_config
Oct 24 16:35:27 btc-apps electrs[3343]:    4: electrs::index::Index::sync
Oct 24 16:35:27 btc-apps electrs[3343]:    5: electrs::server::run
Oct 24 16:35:27 btc-apps electrs[3343]: note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Oct 24 16:35:27 btc-apps electrs[3343]: [2025-10-24T14:35:27.623Z INFO  electrs::db] closing DB at /home/electrs/.electrs/index_db/bitcoin
Oct 24 16:35:28 btc-apps electrs[3343]: [2025-10-24T14:35:28.640Z DEBUG electrs::p2p] closing p2p_loop thread: no more requests to handle
Oct 24 16:35:28 btc-apps electrs[3343]: [2025-10-24T14:35:28.640Z DEBUG electrs::p2p] closing p2p_send thread: no more messages to send
Oct 24 16:35:28 btc-apps electrs[3343]: [2025-10-24T14:35:28.640Z DEBUG electrs::p2p] closing p2p_recv thread: connection closed
Oct 24 16:35:28 btc-apps systemd[1]: electrs.service: Main process exited, code=exited, status=101/n/a
Oct 24 16:35:28 btc-apps systemd[1]: electrs.service: Failed with result 'exit-code'.
Oct 24 16:35:28 btc-apps systemd[1]: electrs.service: Consumed 15min 19.235s CPU time, 3.1G memory peak.

DB LOG file attached.

End state

After more than one day and repeated restarts it seems to have finally finished. Current size of index on disk is ~211GB, which doesn't appear very compacted. Reporting in case this is of any relevance and off the expected behavior.

Once the indexing/compactification finished, every attempt by the client to connect causes a crash of the server, which gets restarted by systemd. The error is the same as above

Oct 25 09:18:48 btc-apps electrs[12876]: DB scan failed: Error { message: "IO error: While open a file for random read: /home/electrs/.electrs/index_db/bitcoin/001274.sst: Too many open files" }

leilerg avatar Oct 25 '25 07:10 leilerg

Update

Tried the initial sync with v0.10.9. First attempt used same VM specs as above and it was not successful. The error was different though - I was seeing "Out of memory" errors in the system log. Otherwise it seemed to proceed as you would expect. Butthe final DB was still very large.

I then increased the RAM of the VM to 8GB and everything completed without issues. After compacting the DB size is ~54GB and I can connect to the server. Everything works as it should.

Observations

  • There seems to be something broken with v0.10.10 for sure. Did not try with more resources though.
  • Given my experience with v0.10.9 I believe there's no mechanism currently in place for a graceful resume of the initial sync, if this gets interrupted. The data seems to copied multiple times, each time it's restarted.
  • Likewise, there's no mechanism to guard against excessive memory use. Maybe it's not meant to be ran on older/low-spec hardware, and so be it.

leilerg avatar Oct 28 '25 13:10 leilerg

Could you please also try 0.11.0 release?

romanz avatar Nov 23 '25 15:11 romanz 

Dec 06 13:44:52 btc-apps electrs[5112]: Error: electrs failed
Dec 06 13:44:52 btc-apps electrs[5112]: Caused by:
Dec 06 13:44:52 btc-apps electrs[5112]:     0: sync failed
Dec 06 13:44:52 btc-apps electrs[5112]:     1: failed to get block 000000000000000001ab06bbe96b8f100a09331db9953b33d619febb8ef425f2
Dec 06 13:44:52 btc-apps electrs[5112]:     2: receiving on an empty and disconnected channel
Dec 06 13:44:52 btc-apps electrs[5112]: Stack backtrace:
Dec 06 13:44:52 btc-apps electrs[5112]:    0: std::sys::backtrace::__rust_begin_short_backtrace
Dec 06 13:44:52 btc-apps electrs[5112]:    1: core::ops::function::FnOnce::call_once{{vtable.shim}}
Dec 06 13:44:52 btc-apps electrs[5112]:    2: std::sys::pal::unix::thread::Thread::new::thread_start
Dec 06 13:44:52 btc-apps electrs[5112]:    3: <unknown>
Dec 06 13:44:52 btc-apps electrs[5112]:    4: <unknown>
Dec 06 13:44:52 btc-apps systemd[1]: electrs.service: Main process exited, code=exited, status=1/FAILURE
Dec 06 13:44:52 btc-apps systemd[1]: electrs.service: Failed with result 'exit-code'.
Dec 06 13:44:52 btc-apps systemd[1]: electrs.service: Consumed 53min 42.813s CPU time, 6.3G memory peak.
Dec 06 13:45:07 btc-apps systemd[1]: electrs.service: Scheduled restart job, restart counter is at 1.
Dec 06 13:45:07 btc-apps systemd[1]: Started electrs.service - Electrs Electrum Server.
Dec 06 13:45:07 btc-apps electrs[5975]: Starting electrs 0.11.0 on x86_64 linux with Config { network: Bitcoin, db_path: "/home/electrs/.electrs/index_db/bitcoin", db_log_dir: None, db_parallelism: 2, daemon_auth: UserPass("electrs", "<sensitive>"), daemon_rpc_addr: 10.1.1.10:8332, daemon_p2p_addr: 10.1.1.10:8333, electrum_rpc_addr: 10.1.1.11:50001, monitoring_addr: 127.0.0.1:4224, wait_duration: 10s, jsonrpc_timeout: 15s, index_batch_size: 10, index_lookup_limit: None, reindex_last_blocks: 0, auto_reindex: true, ignore_mempool: false, sync_once: false, skip_block_download_wait: false, disable_electrum_rpc: false, server_banner: "Welcome to electrs 0.11.0 (Electrum Rust Server)!", signet_magic: f9beb4d9 }
Dec 06 13:45:07 btc-apps electrs[5975]: [2025-12-06T12:45:07.319Z INFO  electrs::metrics::metrics_impl] serving Prometheus metrics on 127.0.0.1:4224
Dec 06 13:45:07 btc-apps electrs[5975]: [2025-12-06T12:45:07.319Z INFO  electrs::server] serving Electrum RPC on 10.1.1.11:50001
Dec 06 13:45:07 btc-apps electrs[5975]: [2025-12-06T12:45:07.469Z INFO  electrs::db] "/home/electrs/.electrs/index_db/bitcoin": 169 SST files, 13.696701131 GB, 1.142548694 Grows
Dec 06 13:45:07 btc-apps electrs[5975]: [2025-12-06T12:45:07.997Z INFO  electrs::chain] loading 452580 headers, tip=000000000000000000d7e67e7a09191d43c72307196b1491bd928dbe16ef468f
Dec 06 13:45:08 btc-apps electrs[5975]: [2025-12-06T12:45:08.531Z INFO  electrs::chain] chain updated: tip=000000000000000000d7e67e7a09191d43c72307196b1491bd928dbe16ef468f, height=452580
Dec 06 13:45:09 btc-apps electrs[5975]: [2025-12-06T12:45:09.577Z INFO  electrs::daemon] waiting for 12 blocks to download
Dec 06 13:45:30 btc-apps electrs[5975]: [2025-12-06T12:45:30.187Z INFO  electrs::daemon] waiting for 11 blocks to download
Dec 06 13:45:36 btc-apps electrs[5975]: [2025-12-06T12:45:36.789Z INFO  electrs::daemon] waiting for 10 blocks to download
Dec 06 13:45:42 btc-apps electrs[5975]: [2025-12-06T12:45:42.772Z INFO  electrs::daemon] waiting for 10 blocks to download
Dec 06 13:45:54 btc-apps electrs[5975]: [2025-12-06T12:45:54.949Z INFO  electrs::daemon] waiting for 9 blocks to download
Dec 06 13:45:56 btc-apps electrs[5975]: [2025-12-06T12:45:56.415Z INFO  electrs::daemon] waiting for 9 blocks to download
Dec 06 13:46:06 btc-apps electrs[5975]: [2025-12-06T12:46:06.686Z INFO  electrs::daemon] waiting for 8 blocks to download
Dec 06 13:46:08 btc-apps electrs[5975]: [2025-12-06T12:46:08.306Z INFO  electrs::daemon] waiting for 8 blocks to download
Dec 06 13:46:13 btc-apps electrs[5975]: [2025-12-06T12:46:13.561Z INFO  electrs::daemon] waiting for 7 blocks to download
Dec 06 13:46:24 btc-apps electrs[5975]: [2025-12-06T12:46:24.675Z INFO  electrs::daemon] waiting for 6 blocks to download
Dec 06 13:46:33 btc-apps electrs[5975]: [2025-12-06T12:46:33.804Z INFO  electrs::daemon] waiting for 5 blocks to download
Dec 06 13:46:51 btc-apps electrs[5975]: [2025-12-06T12:46:51.581Z INFO  electrs::daemon] waiting for 4 blocks to download
Dec 06 13:46:59 btc-apps electrs[5975]: [2025-12-06T12:46:59.619Z INFO  electrs::daemon] waiting for 3 blocks to download
Dec 06 13:47:02 btc-apps electrs[5975]: [2025-12-06T12:47:02.654Z INFO  electrs::daemon] waiting for 3 blocks to download
Dec 06 13:47:04 btc-apps electrs[5975]: [2025-12-06T12:47:04.186Z INFO  electrs::daemon] waiting for 3 blocks to download
Dec 06 13:47:07 btc-apps electrs[5975]: [2025-12-06T12:47:07.957Z INFO  electrs::daemon] waiting for 2 blocks to download
Dec 06 13:47:29 btc-apps electrs[5975]: [2025-12-06T12:47:29.462Z INFO  electrs::daemon] waiting for 1 blocks to download
Dec 06 13:47:32 btc-apps electrs[5975]: [2025-12-06T12:47:32.703Z INFO  electrs::daemon] waiting for 1 blocks to download
Dec 06 13:47:37 btc-apps electrs[5975]: [2025-12-06T12:47:37.411Z INFO  electrs::daemon] waiting for 1 blocks to download
Dec 06 13:47:43 btc-apps electrs[5975]: [2025-12-06T12:47:43.359Z INFO  electrs::daemon] waiting for 1 blocks to download
Dec 06 13:47:45 btc-apps electrs[5975]: [2025-12-06T12:47:45.141Z INFO  electrs::daemon] waiting for 1 blocks to download
Dec 06 13:47:48 btc-apps electrs[5975]: [2025-12-06T12:47:48.488Z INFO  electrs::daemon] waiting for 1 blocks to download
Dec 06 13:47:52 btc-apps electrs[5975]: [2025-12-06T12:47:52.846Z INFO  electrs::daemon] waiting for 1 blocks to download
Dec 06 13:48:08 btc-apps electrs[5975]: [2025-12-06T12:48:08.741Z INFO  electrs::index] indexing 2000 blocks: [452581..454580]

This happened now 3 times (restart counter at 3) and hasn't finished yet. It's continuing to churn though and appears to be recovering from the block it stopped. Can be seen above as well, restarts indexing from blocks [452581..454580].

Meanwhile, another error has appeared:

Dec 06 19:27:43 btc-apps electrs[7000]: [2025-12-06T18:27:43.557Z INFO  electrs::chain] chain updated: tip=00000000000000000005e73d6374615c8348cc8d2d39b2f55ca687cbad68f4fd, height=723490
Dec 06 19:27:43 btc-apps electrs[7000]: [2025-12-06T18:27:43.573Z INFO  electrs::index] indexing 2000 blocks: [723491..725490]
Dec 06 19:27:54 btc-apps kernel: [TTM] Buffer eviction failed
Dec 06 19:28:09 btc-apps kernel: [TTM] Buffer eviction failed
Dec 06 19:28:24 btc-apps kernel: [TTM] Buffer eviction failed
Dec 06 19:28:40 btc-apps kernel: [TTM] Buffer eviction failed

But the kernel issue does not seem to be crucial as the indexing continues after the error is reported about a dozen times.

 

Let me know if there's any specific test you'd want me to run.

leilerg avatar Dec 06 '25 18:12 leilerg

Update: The Buffer eviction failed issue happens also with 0.10.9, so not directly related to electrs. Proxmox issues, as it turns out.

leilerg avatar Dec 07 '25 09:12 leilerg