mina
mina copied to clipboard
MinaProtocol
Devnet Archive cannot load block 400187 3NLgcLnqqftWqurJPT3zahMNSQsHGPKZwP842saU1PuSgsHpUXKJ.json
Preliminary Checks
- [x] This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/MinaProtocol/mina/issues
- [x] This issue is not a question, feature request, RFC, or anything other than a bug report. Please post those things in GitHub Discussions: https://github.com/MinaProtocol/mina/discussions
Description
mina-archive -version Commit 6929a7ecd1b5cf977ceb674baa6e32764576c109
mina-archive fails to load devnet block 400187 3NLgcLnqqftWqurJPT3zahMNSQsHGPKZwP842saU1PuSgsHpUXKJ
Steps to Reproduce
- Restore latest backup
- Run missing blocks auditor
- Try to load missing block 400187 via
mina-archive-blocks --precomputed ...
Expected Result
Block will load to the archive db.
Actual Result
Block cannot be loaded, may crash mina-archive.
Daemon version
mina-archive -version Commit 6929a7ecd1b5cf977ceb674baa6e32764576c109
How frequently do you see this issue?
Rarely
What is the impact of this issue on your ability to run a node?
Low
Status
mina-archive-blocks --precomputed --archive-uri "$1" "$2"
Most attempts just "Error when adding block" - finally crashed the archive service:
Dec 03 19:46:03 nym3 mina-archive[820354]: {"timestamp":"2025-12-04 00:46:03.286111Z","level":"Info","source":{"module":"Archive_lib__Processor","location":>
Dec 03 19:46:13 nym3 mina-archive[820354]: (monitor.ml.Error
Dec 03 19:46:13 nym3 mina-archive[820354]: ("Async was unable to add a file descriptor to its table of open file descriptors"
Dec 03 19:46:13 nym3 mina-archive[820354]: (file_descr 8)
Dec 03 19:46:13 nym3 mina-archive[820354]: (error
Dec 03 19:46:13 nym3 mina-archive[820354]: "Attempt to register a file descriptor with Async that Async believes it is already managing.")
Dec 03 19:46:13 nym3 mina-archive[820354]: (backtrace
Dec 03 19:46:13 nym3 mina-archive[820354]: ("Raised by primitive operation at Base__Backtrace.get in file \"src/backtrace.ml\", line 10, characters 2-48"
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_unix__Raw_scheduler.create_fd in file \"src/raw_scheduler.ml\", line 109, characters 66-84"
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_unix__Fd.create in file \"src/fd.ml\" (inlined), line 40, characters 2-112"
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Caqti_async.System_unix.Unix.wrap_fd in file \"caqti-async/lib/caqti_async.ml\", line 186, chara>
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Caqti_driver_postgresql.Connect_functor.Make_connection_base.reset.(fun) in file \"caqti-driver->
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_kernel__Deferred0.bind.(fun) in file \"src/deferred0.ml\", line 54, characters 64-69"
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_kernel__Job_queue.run_job in file \"src/job_queue.ml\" (inlined), line 128, characters 2-5"
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_kernel__Job_queue.run_jobs in file \"src/job_queue.ml\", line 169, characters 6-47"
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_kernel__Scheduler1.run_jobs in file \"src/scheduler1.ml\", line 335, characters 8-51"
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_kernel__Scheduler.run_cycle.run_jobs in file \"src/scheduler.ml\", line 173, characters 10>
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_kernel__Scheduler.run_cycle in file \"src/scheduler.ml\", line 181, characters 2-12"
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_unix__Raw_scheduler.have_lock_do_cycle in file \"src/raw_scheduler.ml\", line 631, charact>
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_unix__Raw_scheduler.be_the_scheduler.loop in file \"src/raw_scheduler.ml\", line 875, char>
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_unix__Raw_scheduler.be_the_scheduler in file \"src/raw_scheduler.ml\", line 879, character>
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_command.in_async.(fun) in file \"async_command/src/async_command.ml\", line 75, characters>
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Core_kernel__Command.For_unix.run.(fun) in file \"src/command.ml\", line 2453, characters 8-238"
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Base__Exn.handle_uncaught_aux in file \"src/exn.ml\", line 111, characters 6-10"
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Dune__exe__Archive in file \"src/app/archive/archive.ml\", line 15, characters 6-95"))
Dec 03 19:46:13 nym3 mina-archive[820354]: (scheduler
Dec 03 19:46:13 nym3 mina-archive[820354]: ((mutex
Dec 03 19:46:13 nym3 mina-archive[820354]: ((id_of_thread_holding_lock 0) (num_using_blocker 0) (blocker ())))
Dec 03 19:46:13 nym3 mina-archive[820354]: (is_running true) (have_called_go true)
Dec 03 19:46:13 nym3 mina-archive[820354]: (fds_whose_watching_has_changed ())
Dec 03 19:46:13 nym3 mina-archive[820354]: (file_descr_watcher
Dec 03 19:46:13 nym3 mina-archive[820354]: (Epoll
Dec 03 19:46:13 nym3 mina-archive[820354]: ((timerfd 5)
Dec 03 19:46:13 nym3 mina-archive[820354]: (epoll
Dec 03 19:46:13 nym3 mina-archive[820354]: (In_use
Dec 03 19:46:13 nym3 mina-archive[820354]: ((epollfd 6)
Dec 03 19:46:13 nym3 mina-archive[820354]: (flags_by_fd
Dec 03 19:46:13 nym3 mina-archive[820354]: ((num_keys 65536) (alist ((7 (in)) (3 (in)) (5 (in et))))))
Dec 03 19:46:13 nym3 mina-archive[820354]: (max_ready_events 256) (num_ready_events 0) (ready_events ()))))
Dec 03 19:46:13 nym3 mina-archive[820354]: (handle_fd_read_ready <fun>) (handle_fd_write_ready <fun>))))
Dec 03 19:46:13 nym3 mina-archive[820354]: (time_spent_waiting_for_io 642509882619)
Dec 03 19:46:13 nym3 mina-archive[820354]: (fd_by_descr
Dec 03 19:46:13 nym3 mina-archive[820354]: ((3
Dec 03 19:46:13 nym3 mina-archive[820354]: ((file_descr 3) (info interruptor_pipe_read) (kind Fifo)
Dec 03 19:46:13 nym3 mina-archive[820354]: (supports_nonblock true) (have_set_nonblock true)
Dec 03 19:46:13 nym3 mina-archive[820354]: (state (Open Empty))
Dec 03 19:46:13 nym3 mina-archive[820354]: (watching
Dec 03 19:46:13 nym3 mina-archive[820354]: ((read (Watch_repeatedly "<Pool.Pointer.t: 0x00000001>" Empty))
Dec 03 19:46:13 nym3 mina-archive[820354]: (write Not_watching)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (watching_has_changed false) (num_active_syscalls 1)
Dec 03 19:46:13 nym3 mina-archive[820354]: (close_finished Empty)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (4
Dec 03 19:46:13 nym3 mina-archive[820354]: ((file_descr 4) (info interruptor_pipe_write) (kind Fifo)
Dec 03 19:46:13 nym3 mina-archive[820354]: (supports_nonblock true) (have_set_nonblock true)
Dec 03 19:46:13 nym3 mina-archive[820354]: (state (Open Empty))
Dec 03 19:46:13 nym3 mina-archive[820354]: (watching ((read Not_watching) (write Not_watching)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (watching_has_changed false) (num_active_syscalls 0)
Dec 03 19:46:13 nym3 mina-archive[820354]: (close_finished Empty)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (7
Dec 03 19:46:13 nym3 mina-archive[820354]: ((file_descr 7) (info ((type_ tcp) (listening_on 0.0.0.0:3086)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (kind (Socket Passive)) (supports_nonblock true)
Dec 03 19:46:13 nym3 mina-archive[820354]: (have_set_nonblock true) (state (Open Empty))
Dec 03 19:46:13 nym3 mina-archive[820354]: (watching ((read (Watch_once Empty)) (write Not_watching)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (watching_has_changed false) (num_active_syscalls 1)
Dec 03 19:46:13 nym3 mina-archive[820354]: (close_finished Empty)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (8
Dec 03 19:46:13 nym3 mina-archive[820354]: ((file_descr 8) (info "Caqti_async file descriptor")
Dec 03 19:46:13 nym3 mina-archive[820354]: (kind (Socket Active)) (supports_nonblock true)
Dec 03 19:46:13 nym3 mina-archive[820354]: (have_set_nonblock false) (state (Open Empty))
Dec 03 19:46:13 nym3 mina-archive[820354]: (watching ((read Not_watching) (write Not_watching)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (watching_has_changed false) (num_active_syscalls 0)
Dec 03 19:46:13 nym3 mina-archive[820354]: (close_finished Empty)))))
Dec 03 19:46:13 nym3 mina-archive[820354]: (timerfd (5)) (timerfd_set_at (2025-12-03 19:46:13.091352576-05:00))
Dec 03 19:46:13 nym3 mina-archive[820354]: (scheduler_thread_id 0)
Dec 03 19:46:13 nym3 mina-archive[820354]: (interruptor
Dec 03 19:46:13 nym3 mina-archive[820354]: ((pipe
Dec 03 19:46:13 nym3 mina-archive[820354]: ((read
Dec 03 19:46:13 nym3 mina-archive[820354]: ((file_descr 3) (info interruptor_pipe_read) (kind Fifo)
Dec 03 19:46:13 nym3 mina-archive[820354]: (supports_nonblock true) (have_set_nonblock true)
Dec 03 19:46:13 nym3 mina-archive[820354]: (state (Open Empty))
Dec 03 19:46:13 nym3 mina-archive[820354]: (watching
Dec 03 19:46:13 nym3 mina-archive[820354]: ((read (Watch_repeatedly "<Pool.Pointer.t: 0x00000001>" Empty))
Dec 03 19:46:13 nym3 mina-archive[820354]: (write Not_watching)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (watching_has_changed false) (num_active_syscalls 1)
Dec 03 19:46:13 nym3 mina-archive[820354]: (close_finished Empty)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (write
Dec 03 19:46:13 nym3 mina-archive[820354]: ((file_descr 4) (info interruptor_pipe_write) (kind Fifo)
Dec 03 19:46:13 nym3 mina-archive[820354]: (supports_nonblock true) (have_set_nonblock true)
Dec 03 19:46:13 nym3 mina-archive[820354]: (state (Open Empty))
Dec 03 19:46:13 nym3 mina-archive[820354]: (watching ((read Not_watching) (write Not_watching)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (watching_has_changed false) (num_active_syscalls 0)
Dec 03 19:46:13 nym3 mina-archive[820354]: (close_finished Empty)))))
Dec 03 19:46:13 nym3 mina-archive[820354]: (already_interrupted false) (clearbuffer <opaque>)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (signal_manager
Dec 03 19:46:13 nym3 mina-archive[820354]: ((handlers_by_signal ((-8 ((bag ()))))) (delivered <opaque>)
Dec 03 19:46:13 nym3 mina-archive[820354]: (thread_safe_notify_signal_delivered <fun>)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (thread_pool
Dec 03 19:46:13 nym3 mina-archive[820354]: ((id 0) (state In_use)
Dec 03 19:46:13 nym3 mina-archive[820354]: (finished
Dec 03 19:46:13 nym3 mina-archive[820354]: ((value ()) (num_waiting 0) (mutex <opaque>) (full <opaque>)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (mutex
Dec 03 19:46:13 nym3 mina-archive[820354]: ((id_of_thread_holding_lock -1) (num_using_blocker 0) (blocker ())))
Dec 03 19:46:13 nym3 mina-archive[820354]: (default_priority 0) (max_num_threads 50) (cpu_affinity Inherit)
Dec 03 19:46:13 nym3 mina-archive[820354]: (num_threads 2) (thread_creation_failure_lockout 1s)
Dec 03 19:46:13 nym3 mina-archive[820354]: (last_thread_creation_failure ())
Dec 03 19:46:13 nym3 mina-archive[820354]: (thread_by_id
Dec 03 19:46:13 nym3 mina-archive[820354]: ((1
Dec 03 19:46:13 nym3 mina-archive[820354]: ((name shutdown) (thread_id (1)) (priority 0) (state Available)
Dec 03 19:46:13 nym3 mina-archive[820354]: (unfinished_work 0)
Dec 03 19:46:13 nym3 mina-archive[820354]: (work_queue
Dec 03 19:46:13 nym3 mina-archive[820354]: ((ev_q ()) (maxsize 1) (mutex <opaque>) (not_empty <opaque>)
Dec 03 19:46:13 nym3 mina-archive[820354]: (not_full <opaque>) (finally <fun>)))))
Dec 03 19:46:13 nym3 mina-archive[820354]: (2
Dec 03 19:46:13 nym3 systemd[695]: mina-archive.service: Main process exited, code=exited, status=1/FAILURE
Dec 03 19:46:13 nym3 mina-archive[820354]: ((name close) (thread_id (2)) (priority 0) (state Available)
Dec 03 19:46:13 nym3 mina-archive[820354]: (unfinished_work 0)
Dec 03 19:46:13 nym3 mina-archive[820354]: (work_queue
Dec 03 19:46:13 nym3 mina-archive[820354]: ((ev_q ()) (maxsize 1) (mutex <opaque>) (not_empty <opaque>)
Dec 03 19:46:13 nym3 mina-archive[820354]: (not_full <opaque>) (finally <fun>)))))))
Dec 03 19:46:13 nym3 mina-archive[820354]: (available_threads
Dec 03 19:46:13 nym3 mina-archive[820354]: (((name close) (thread_id (2)) (priority 0) (state Available)
Dec 03 19:46:13 nym3 mina-archive[820354]: (unfinished_work 0)
Dec 03 19:46:13 nym3 mina-archive[820354]: (work_queue
Dec 03 19:46:13 nym3 mina-archive[820354]: ((ev_q ()) (maxsize 1) (mutex <opaque>) (not_empty <opaque>)
Dec 03 19:46:13 nym3 mina-archive[820354]: (not_full <opaque>) (finally <fun>))))
Dec 03 19:46:13 nym3 mina-archive[820354]: ((name shutdown) (thread_id (1)) (priority 0) (state Available)
Dec 03 19:46:13 nym3 mina-archive[820354]: (unfinished_work 0)
Dec 03 19:46:13 nym3 mina-archive[820354]: (work_queue
Dec 03 19:46:13 nym3 mina-archive[820354]: ((ev_q ()) (maxsize 1) (mutex <opaque>) (not_empty <opaque>)
Dec 03 19:46:13 nym3 mina-archive[820354]: (not_full <opaque>) (finally <fun>))))))
Dec 03 19:46:13 nym3 mina-archive[820354]: (work_queue ()) (unfinished_work 0) (num_work_completed 5)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (handle_thread_pool_stuck <fun>)
Dec 03 19:46:13 nym3 mina-archive[820354]: (next_tsc_calibration 173303125748506281)
Dec 03 19:46:13 nym3 mina-archive[820354]: (kernel_scheduler
Dec 03 19:46:13 nym3 mina-archive[820354]: ((check_access ())
Dec 03 19:46:13 nym3 mina-archive[820354]: (job_pool
Dec 03 19:46:13 nym3 mina-archive[820354]: ((slots_per_tuple 3) (capacity 4) (length 2) (next_id 202)
Dec 03 19:46:13 nym3 mina-archive[820354]: (first_free (Free 9)) (dummy (<opaque>))))
Dec 03 19:46:13 nym3 mina-archive[820354]: (normal_priority_jobs
Dec 03 19:46:13 nym3 mina-archive[820354]: ((num_jobs_run 112018) (jobs_left_this_cycle 467) (jobs <opaque>)
Dec 03 19:46:13 nym3 mina-archive[820354]: (mask 3) (front 2) (length 0)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (low_priority_jobs
Dec 03 19:46:13 nym3 mina-archive[820354]: ((num_jobs_run 8) (jobs_left_this_cycle 500) (jobs <opaque>) (mask 0)
Dec 03 19:46:13 nym3 mina-archive[820354]: (front 0) (length 0)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (very_low_priority_workers ())
Dec 03 19:46:13 nym3 mina-archive[820354]: (main_execution_context
Dec 03 19:46:13 nym3 mina-archive[820354]: ((monitor
Dec 03 19:46:13 nym3 mina-archive[820354]: (((name main) (here ()) (id 1) (has_seen_error false)
Dec 03 19:46:13 nym3 mina-archive[820354]: (is_detached false))))
Dec 03 19:46:13 nym3 mina-archive[820354]: (priority Normal) (local_storage ()) (tid 0) (backtrace_history ())))
Dec 03 19:46:13 nym3 mina-archive[820354]: (current_execution_context
Dec 03 19:46:13 nym3 mina-archive[820354]: ((monitor
Dec 03 19:46:13 nym3 mina-archive[820354]: (((name "") (here ()) (id 1620) (has_seen_error false)
Dec 03 19:46:13 nym3 mina-archive[820354]: (is_detached true))))
Dec 03 19:46:13 nym3 mina-archive[820354]: (priority Normal) (local_storage ()) (tid 0) (backtrace_history ())))
Dec 03 19:46:13 nym3 mina-archive[820354]: (uncaught_exn ()) (cycle_count 10880)
Dec 03 19:46:13 nym3 mina-archive[820354]: (cycle_start "2025-12-04 00:46:13.079433623Z") (in_cycle true)
Dec 03 19:46:13 nym3 mina-archive[820354]: (run_every_cycle_start (<fun>)) (run_every_cycle_end (<fun>))
Dec 03 19:46:13 nym3 systemd[695]: mina-archive.service: Failed with result 'exit-code'.
Dec 03 19:46:13 nym3 mina-archive[820354]: (last_cycle_time 97.002us) (last_cycle_num_jobs 5)
Dec 03 19:46:13 nym3 mina-archive[820354]: (total_cycle_time 2.267359177s)
Dec 03 19:46:13 nym3 mina-archive[820354]: (time_source (wall_clock (now 2025-12-03T19:46:13-0500)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (external_jobs
Dec 03 19:46:13 nym3 mina-archive[820354]: ((length 0) (front ((value None) (next <opaque>)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (back ((value None) (next <opaque>))) (unused_elts None)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (thread_safe_external_job_hook <fun>) (job_queued_hook ())
Dec 03 19:46:13 nym3 mina-archive[820354]: (event_added_hook ()) (yield <opaque>)
Dec 03 19:46:13 nym3 mina-archive[820354]: (yield_until_no_jobs_remain <opaque>) (check_invariants false)
Dec 03 19:46:13 nym3 mina-archive[820354]: (max_num_jobs_per_priority_per_cycle 500) (record_backtraces false)
Dec 03 19:46:13 nym3 mina-archive[820354]: (on_start_of_cycle <fun>) (on_end_of_cycle <fun>)
Dec 03 19:46:13 nym3 mina-archive[820354]: (long_jobs_last_cycle ()) (cycle_started false)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (have_lock_do_cycle ()) (max_inter_cycle_timeout 50ms)
Dec 03 19:46:13 nym3 mina-archive[820354]: (min_inter_cycle_timeout 0s) (initialized_at <opaque>))))
Dec 03 19:46:13 nym3 mina-archive[820354]: ("Raised at Base__Error.raise in file \"src/error.ml\" (inlined), line 8, characters 14-30"
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Caqti_async.Fiber.finally.(fun) in file \"caqti-async/lib/caqti_async.ml\", line 44, characters 39>
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_kernel__Deferred1.M.map.(fun) in file \"src/deferred1.ml\", line 17, characters 40-45"
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_kernel__Job_queue.run_job in file \"src/job_queue.ml\" (inlined), line 128, characters 2-5"
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_kernel__Job_queue.run_jobs in file \"src/job_queue.ml\", line 169, characters 6-47"))
Dec 03 19:46:13 nym3 systemd[695]: mina-archive.service: Consumed 3.102s CPU time.
Additional information
This was happening in my devnet node; I restored the latest archive backup from o1 and it is also missing the same block.
There are a couple of other blocks missing their parent hash; I did not try to load them but since they are also missing from o1 backup, they may have similar issues.
400188 | 3NLgcLnqqftWqurJPT3zahMNSQsHGPKZwP842saU1PuSgsHpUXKJ
470950 | 3NLXw8tC611162GJRQtnEdhC8PiMKBFrQAfzaCFJdRQZnKhQNVNB
474527 | 3NLtAJDxmaH5wgFu5qSymZWfjqVnLt34aQJVe4b3YNqgA65XWtdg
All blocks you mentioned above are inserted in db:
archive=# select id,height,chain_status from blocks where state_hash in ('3NLgcLnqqftWqurJPT3zahMNSQsHGPKZwP842saU1PuSgsHpUXKJ',
'3NLXw8tC611162GJRQtnEdhC8PiMKBFrQAfzaCFJdRQZnKhQNVNB',
'3NLtAJDxmaH5wgFu5qSymZWfjqVnLt34aQJVe4b3YNqgA65XWtdg');
id | height | chain_status
--------+--------+--------------
470388 | 400187 | canonical
588275 | 474526 | canonical
583220 | 470949 | canonical
(3 rows)
I used dump from : https://storage.googleapis.com/mina-archive-dumps/devnet-archive-dump-2025-12-09_0900.sql.tar.gz
missing blocks auditor is also clean:
{"timestamp":"2025-12-09 10:19:10.869014Z","level":"Info","source":{"module":"Dune__exe__Missing_blocks_auditor","location":"File \"src/app/missing_blocks_auditor/missing_blocks_auditor.ml\", line 32, characters 6-17"},"message":"Successfully created Caqti pool for Postgresql","metadata":{}}
{"timestamp":"2025-12-09 10:19:10.869038Z","level":"Info","source":{"module":"Dune__exe__Missing_blocks_auditor","location":"File \"src/app/missing_blocks_auditor/missing_blocks_auditor.ml\", line 33, characters 6-17"},"message":"Querying missing blocks","metadata":{}}
{"timestamp":"2025-12-09 10:19:10.892110Z","level":"Info","source":{"module":"Dune__exe__Missing_blocks_auditor","location":"File \"src/app/missing_blocks_auditor/missing_blocks_auditor.ml\", line 67, characters 20-31"},"message":"There are no missing blocks in the archive db","metadata":{}}
{"timestamp":"2025-12-09 10:19:10.892136Z","level":"Info","source":{"module":"Dune__exe__Missing_blocks_auditor","location":"File \"src/app/missing_blocks_auditor/missing_blocks_auditor.ml\", line 92, characters 6-17"},"message":"Querying for gaps in chain statuses","metadata":{}}
{"timestamp":"2025-12-09 10:19:10.893579Z","level":"Info","source":{"module":"Dune__exe__Missing_blocks_auditor","location":"File \"src/app/missing_blocks_auditor/missing_blocks_auditor.ml\", line 123, characters 8-19"},"message":"There are no gaps in the chain statuses","metadata":{}}
{"timestamp":"2025-12-09 10:19:12.009281Z","level":"Info","source":{"module":"Dune__exe__Missing_blocks_auditor","location":"File \"src/app/missing_blocks_auditor/missing_blocks_auditor.ml\", line 150, characters 8-19"},"message":"Length of canonical chain is 478758 blocks","metadata":{}}
{"timestamp":"2025-12-09 10:19:12.013289Z","level":"Info","source":{"module":"Dune__exe__Missing_blocks_auditor","location":"File \"src/app/missing_blocks_auditor/missing_blocks_auditor.ml\", line 167, characters 8-19"},"message":"All blocks along the canonical chain have a valid chain status","metadata":{}}
Replayer run is ok as well:
2025-12-09 01:10:20 UTC [Info] Not writing checkpoint file at slot 737760, because not canonical
2025-12-09 01:10:20 UTC [Info] No target epoch ledger hash supplied, not writing output
[2025-12-09T01:10:20+00:00] INFO: Replayer execution completed
[2025-12-09T01:10:20+00:00] INFO: Processing replayer results
[2025-12-09T01:10:20+00:00] INFO: No errors found in replayer log
[2025-12-09T01:10:20+00:00] INFO: Uploading checkpoint: replayer-checkpoint-737206.json -> berkeley-devnet-replayer-checkpoint-2025-12-09.json
I cannot find similar error in our logs.
However, I do remember this error and the issue is in too many connections opened to postgres. Thanks for putting information regarding version you are using as this is known issue for archive node you are running. We fixed above leak on 3.3.0 (PR: https://github.com/MinaProtocol/mina/pull/16704) (https://github.com/MinaProtocol/mina/releases/tag/3.3.0-beta1). This is devnet, so i think it is acceptable to update to beta.
The question remain why our dump didn't have mentioned block too.
Can you provide exact dump which you downloaded from o1 backups ?
I restored devnet-archive-dump-2025-12-03_2300.sql -- but I think I queried the wrong database or something went wrong with the restore and I missed it ... because those blocks are all found in the raw dump.
Sorry for the false alarm; I'll install the 3.3.0 beta to devnet and see if it will reprocess the blocks.
% grep 3NLgcLnqqftWqurJPT3zahMNSQsHGPKZwP842saU1PuSgsHpUXKJ ~/Downloads/devnet-archive-dump-2025-12-03_2300.sql
470388 3NLgcLnqqftWqurJPT3zahMNSQsHGPKZwP842saU1PuSgsHpUXKJ 470385 3NLZdcNBwk7jPS1xi8CZmnTish7Mu4ZitW9i4h9wUiMndXCXnzTh 19 5582 ODloFcOzz6XNWaVdoKgwrO49Mt4wA3bqadA9ak0cAAA= 17796 440701 446347 3 {1,6,6,3,6,6,5,4,5,5,4} 1485370858000001000 jxcRfQwMtPXGPvcT1rXwppHzS8whyKPe61EnSbpMh98CX34PbQz 400187 155848 601708 2 \N 1740749040000 canonical
470389 3NLnQLzTm87UACG8H8dwSiGNpTvKmWSxKFzX1TQsrUQigZ9vV2eX 470388 3NLgcLnqqftWqurJPT3zahMNSQsHGPKZwP842saU1PuSgsHpUXKJ 6 6 BRJULNtc31vcmdtQA56Ic5ct_ijS-7S-NVWU8wVqEAA= 17796 440701 446348 3 {2,6,6,3,6,6,5,4,5,5,4} 1485370858000001000 jwcgbFZEfCgRAPbqyeHr5nY9kGqen3EdsP4eKsLA4s2fdwj68c9 400188 155849 601709 2 \N 1740749220000 canonical
470390 3NLDjqnaR2MLmdec5sJ6WdpZoxescDxQiVd6n1yKzgDPDdNKeX8b 470388 3NLgcLnqqftWqurJPT3zahMNSQsHGPKZwP842saU1PuSgsHpUXKJ 18 5680 f50rrlAj1LRU-PByEEPniQr0xFykw14QOOUl1HxNAAA= 17796 440701 446348 3 {2,6,6,3,6,6,5,4,5,5,4} 1485370858000001000 jwMSgzF8aah4ZjXxDXFHD6HqMdAbxHcnexFUdN9SWvRWUXeY45V 400188 155849 601709 2 \N 1740749220000 orphaned
% grep 3NLXw8tC611162GJRQtnEdhC8PiMKBFrQAfzaCFJdRQZnKhQNVNB ~/Downloads/devnet-archive-dump-2025-12-03_2300.sql
583220 3NLXw8tC611162GJRQtnEdhC8PiMKBFrQAfzaCFJdRQZnKhQNVNB 583218 3NLA9QGqespStw9yzSZ65Ntf2dbT1quX5nrX8ws6iDHmirVBBi69 6 6709 Rp_5tJFQOgCUkcfBNF9076UiAzuySVF41UAvgdY4AAA= 20717 536107 540607 3 {6,2,3,5,3,6,4,7,3,3,1} 1534845925000001000 jweqdz2CGQmxdo7aZraWZhNnSZWVe8BRw9DbjELRPTYYMZ8fGVU 470949 277502 723362 2 \N 1762646760000 canonical
583221 3NKbK9A7JNfGnYrgrHMw6pxyuxNSPB71cHyMLkZBsbtSjyYLpoza 583220 3NLXw8tC611162GJRQtnEdhC8PiMKBFrQAfzaCFJdRQZnKhQNVNB 15 293 N5cJHaaH5PfIebXX4nleWBiC_3rWv3U4nVxt-8yfDQA= 20717 536107 540608 3 {6,2,3,5,3,6,4,7,3,3,2} 1534845925000001000 jwR7Wzd6xSQDwAsWtZELX4YKwg3MfDUcexx81pFoYo57ZVfi3sG 470950 277503 723363 2 \N 1762646940000 canonical
% grep 3NLtAJDxmaH5wgFu5qSymZWfjqVnLt34aQJVe4b3YNqgA65XWtdg ~/Downloads/devnet-archive-dump-2025-12-03_2300.sql
588277 3NL2svRQYVg6B4kbDsnLvZnTuBf88KEQ9bSSk4VERaSrK4wcjgme 588275 3NLtAJDxmaH5wgFu5qSymZWfjqVnLt34aQJVe4b3YNqgA65XWtdg 3 3895 oxLQvDF80L6lxM0dCD87Sz1CYy7aMuavK2pGMxANAAA= 20900 541142 545297 3 {2,5,4,4,4,3,5,5,4,1,3} 1537666284000001000 jxPiD359czW5g9jbkEFaY7KgqTq4kPT9taYTUHgW8pUr3XTK1ei 474527 28396729822 2 \N 1763809560000 orphaned
588275 3NLtAJDxmaH5wgFu5qSymZWfjqVnLt34aQJVe4b3YNqgA65XWtdg 588274 3NKieiZKv3bwBfvpWeTAv7RB7cGzqRH8Mp5h1NCJFbsPzEaNm3ZS 9 2851 8XngfEXaUpwjG6rIsq49dIL20ueCNrHyl2IWcds4AAA= 20900 541142 545296 3 {2,5,4,4,4,3,5,5,4,4,3} 1537666284000001000 jwEPH6WfsamLAXA3WJiwsdV9DEjHndb3mpUJzgrS5bbbKQSZEp8 474526 28396729820 2 \N 1763809200000 canonical
588276 3NLDhQBJKYxzhwBzpcZRFKj1Wqn2zNQ1VTeb5dwEDikJy8FYCZfe 588275 3NLtAJDxmaH5wgFu5qSymZWfjqVnLt34aQJVe4b3YNqgA65XWtdg 2 537 MSKGmRsuY6Y_V8kFZZkjVicga_DDenbm275X_GWIAAA= 20900 541142 545297 3 {2,5,4,4,4,3,5,5,4,1,3} 1537666284000001000 jxQQoEwERbqjbUsW6yofXouMLeA15K84eB3Awk8611Y1vfZwkGS 474527 28396729822 2 \N 1763809560000 canonical
It is a bad issue and was reported incorrectly (the part about blocks missing from backup of archive db); however, I don't think there is a devnet build that corrects this issue? I've tried installing beta, but...
E: Version '3.3.0-beta1-5b0a889' for 'mina-archive-devnet' was not found
3.2.1 was a hot release and we only released for mainnet, this is something we should fix by having a release for devnet.
For 3.3.0, we release alpha for devnet and beta for mainnet. So I suggest you try this release out? https://github.com/MinaProtocol/mina/releases/tag/3.3.0-alpha1
They should be the same code being built except some commit are here to help prepare release notes.