steem Accelerating block replay speed

I have replaying blocks on version 0.22.1. with latest block snapshot which is providing from steem developer website. it is first time to do this for me. but it takes more than 3weeks for block replay, it's on going still. Someone give me tips for block replay boosting or anything else would be so much appreciated. if no mira version was faster than mira version, please comment it.

Hardware CPU : Xeon silver 4110 @ 2.10Ghz RAM 32GB SSD 500GB Centos7

When I settled up cmake for steem node as l have done like below;

cmake -DCMAKE_BUILD_TYPE= Release -DENABLE_MIRA=ON -DSTEEM_STATIC_BUILD=ON -DCLEAR_VOTES=ON  -DLOW_MEMORY_NODE=ON ..

config.ini

p2p-endpoint = localhost:2001
webserver-ws-endpoint = {my ip }:8090
webserver-https-endpoint = {my ip }:8090

log-console-appender ={"appender":stderr", "stream":"std_error}
log-file-appender = {"appender":"p2p","file":"logs/p2p/p2p.log"}
log-logger = {"name":"default","level":"info","appender":"stderr"}
log-logger = {"name":"p2p", "level":"warn","appender":"p2p"}
backtrace = yes
plugin = webserver p2p
plugin = account_by_key account_by_key_api condenser_api account_history account_history_api database_api network_broadcast_api rc_api 
account-history-track-account-range = ["youngjnunko","youngjunko"]

history-disable-pruning =1
account-history-rocksdb-path= "blockchain/account-history-rocksdb-storage"
block-data-export-file = NONE
block-log-info-print-interval-seconds = 86400
block-log-info-print-irreversible = 1
block-log-info-print-file = ILOG
shared-file-dir="blockchain"
shared-file-size = 30G
shared-file-full-threshold = 0
shared-file-scale-rate = 0
flush-state-interval= 0
follow-max-feed-size =0
follow-start-feeds =0
market-history-bucket-size=[15,60,300,3600,86400]
market-history-bucket-per-size = 5760
statsd-batchsize =1
tags-start-promoted =0 
tags-skep-startup-update=0
webserver-thread-pool-size = 256
enable-stale-production = false
required-participation = 33
witness-skip-enforce-bandwidth -= true
p2p-parameters ={"listen_endpoint" : "0.0.0.0:0", "accept_incoming_connections" : false, "wait_if_endpoint_is_busy" : true , "private_key" :"0000000000000000000000000000000000000000000000000000000000000000", "desired_number_of_connections":20, "maximum_number_of_connections":200, "peer_connection_retry_timeout":10 , "peer_inactivity_timeout":10, "peer_advertising_disabled": true, ",maximum_number_of_blocks_to_handle_at_one_time":200,"maximum_number_of_sync_blocks_to_prefetch": 2000, "maximum_blocks_per_peer_during_syncing":200, "active_ignored_request_timeout_microseconds": 6000000}

seednode list in txt file.

seed_east.steemit.com:2001
seed-central.steemit.com:2001
seed-west.steemit.com:2001
steem-seed1.abit-more-com:2001
seed.steemd.com:34191
....etc

Please help me guys.

@sgerbino @mvandeberg @theoreticalbts @revflash @Mariusz-Trela @jredbeard @Kiwonik

Oct 31 '19 07:10 johndoe0x

Can you post what some of your logs look like? This configuration should not take that long to replay and I am wondering if your block log is not in the correct location, causing your node to sync the blockchain rather than replay from a local copy.

Oct 31 '19 15:10 mvandeberg

@mvandeberg thanks for reply. When I ran steemd, I add option for --replay-blockchain. and log looks like this. First of all, aware that I download latest blockchain snapshot. logs below.

multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_dynamic_global_object: 
ord_undex_impl:792                    dump_lb_call_counts ] boost::mpl::v_time<steem::chain::by_id, boost::mpl::vector0<mpl_::na>,0>
ord_undex_impl:793                   dump_lb_call_counts ] iterator::lb_call_count(): 0 iterator::lb_prev_call_count(): 0 iterator::lb_no_prev_count():0 iterator::lb_miss_count() : 0

multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_account_object: 
ord_undex_impl:792                    dump_lb_call_counts ] boost::mpl::v_time<steem::chain::by_id, boost::mpl::vector0<mpl_::na>,0>
ord_undex_impl:793                   dump_lb_call_counts ] iterator::lb_call_count(): 0 iterator::lb_prev_call_count(): 0 iterator::lb_no_prev_count():0 iterator::lb_miss_count() : 0
....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_account_metadata_object: 
....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_account_authority_object: 
....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_witness_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_transaction_object: 
....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_block_summary_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_witness_schedule_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_comment_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_comment_content_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_comment_vote_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_witness_vote_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_limit_order_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_feed_history_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_convert_request_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_liquidity_reward_balance_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_operation_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_account_history_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_hardfork_property_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_withdraw_vesting_route_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_owner_authority_history_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_account_recovery_request_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_change_recovery_account_request_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_escrow_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_saving_withdraw_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_decline_voting_rights_request_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_reward_fund_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_vesting_delegation_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_vesting_delegation_expiration_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_pending_required_action_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_pending_optional_action_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_smt_token_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_account_regular_balance_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_account_reward_balance_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_nai_pool_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_smt_token_emissions_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_smt_contribution_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_smt_ico_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_comment_smt_beneficiaries_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_proposal_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_proposal_vote_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_key_lookup_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_rc_resource_param_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_rc_pool_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_rc_account_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_rc_delegation_pool_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_rc_delegation_from_account_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_indel_edge_object: 
.....
multi_index_container.hpp:509 dump_lb_call_counts ] Object rocksdb_rc_outdel_drc_edge_object: 
.....
62.xxxx % 23100000 of 37017766 ( 62500 objects cached using 11M)
.
.
.

It took more than 3 weeks to reach this outcome. Is it different between block reindexing & block replay ??

Nov 01 '19 03:11 johndoe0x

FYI, blog index was empty when I had started to block replay.

Nov 03 '19 05:11 johndoe0x

@MegaSolar

I was able to replay blockchain of my node using latest block_log (https://github.com/steemit/steem/blob/master/doc/exchangequickstart.md) in just 1 day with SKIP_BY_TX_ID, LOW_MEMORY_NODE, MIRA disabled. If you don't need an option to search transactions by trx id (it wasn't necessary for me), then it will fit your needs in full. I wasn't able to sync the FULL node, when MIRA was enabled, the node was shut down due to error at 6 million blocks, so I tried to disable MIRA and it was succeed.

Try to build Steem v0.22.1 with this cmake:

cmake -DCMAKE_BUILD_TYPE=Release -DENABLE_MIRA=OFF -DSKIP_BY_TX_ID=ON -DSTEEM_STATIC_BUILD=ON -DCLEAR_VOTES=ON -DLOW_MEMORY_NODE=ON ..

Server info: i7-6700 Quad-Core 64 GB RAM NVME Ubuntu 18.04.3 LTS

I used to replay blockchain with "cmake -DCMAKE_BUILD_TYPE=Release ..", same config on same machine and it took about 3-4 weeks.

Jan 15 '20 07:01 i3bitcoin

steem steem copied to clipboard

Accelerating block replay speed

When I settled up cmake for steem node as l have done like below;

config.ini

seednode list in txt file.

steem
steem copied to clipboard