stacks-blockchain-api
stacks-blockchain-api copied to clipboard
Published
20 hours ago •
hirosystems
hirosystems
Unable to run a full API node on testnet
Describe the bug
Trying to sync a full API node on testnet.
Port 3999 does not accept requests and stacks tip height does not progress.
Note: I'm also syncing a non API stacks node here http://23.239.9.92:20443/v2/info which is syncing up ok.
Stacks Node Version: "server_version": "stacks-node 2.05.0.1.0 (master:de541f9, release build, linux [x86_64])",
Steps To Reproduce
Following https://docs.hiro.so/get-started/running-api-node but adapting for testnet.
I've tried many configurations - the latest is running a local bitcoin node.
Expected behavior
The stacks tip height to start to increment and curl -sL localhost:3999/v2/info | jq to return node info.
Environment
- OS: Debian 11
- Docker
- Output of
stacks-node versionsee http://172.104.29.143:20443/v2/info Currently
"pox_consensus": "25453ed5b77b48328a6a44e1e97a18e0b43d8e72",
"burn_block_height": 2034651,
"stable_pox_consensus": "0bf090ac81c99ed39adbbd13a341ac2ad5f979a2",
"stable_burn_block_height": 2034644,
"server_version": "stacks-node 2.05.0.1.0 (master:de541f9, release build, linux [x86_64])",
"network_id": 2147483648,
"parent_network_id": 118034699,
"stacks_tip_height": 0,
"stacks_tip": "0000000000000000000000000000000000000000000000000000000000000000",
"stacks_tip_consensus_hash": "0000000000000000000000000000000000000000",
"genesis_chainstate_hash": "74237aa39aa50a83de11a4f53e9d3bb7d43461d1de9873f402e5453ae60bc59b",
"unanchored_tip": null,
"unanchored_seq": null,
"exit_at_block_height": null
Additional context
Contents of .env and config/Config.toml
GIT_TAG=master
PG_HOST=postgres
PG_PORT=5432
PG_USER=postgres
PG_PASSWORD=postgres
PG_DATABASE=postgres
STACKS_CHAIN_ID=2147483648
#STACKS_CHAIN_ID=0x00000001
V2_POX_MIN_AMOUNT_USTX=90000000260
STACKS_CORE_EVENT_PORT=3700
STACKS_CORE_EVENT_HOST=0.0.0.0
STACKS_BLOCKCHAIN_API_PORT=3999
STACKS_BLOCKCHAIN_API_HOST=0.0.0.0
STACKS_BLOCKCHAIN_API_DB=pg
STACKS_CORE_RPC_HOST=stacks-blockchain
STACKS_CORE_RPC_PORT=20443
#BNS_IMPORT_DIR=/bns-data```
```[node]
working_dir = "/root/stacks-node/data"
rpc_bind = "0.0.0.0:20443"
p2p_bind = "0.0.0.0:20444"
bootstrap_node = "047435c194e9b01b3d7f7a2802d6684a3af68d05bbf4ec8f17021980d777691f1d51651f7f1d566532c804da506c117bbf79ad62eea81213ba58f8808b4d9504ad@testnet.stacks.co:20444"
wait_time_for_microblocks = 10000
[[events_observer]]
endpoint = "stacks-blockchain-api:3700"
retry_count = 255
events_keys = ["*"]
[burnchain]
chain = "bitcoin"
mode = "xenon"
peer_host = "172.104.29.143"
username = "blockstack"
password = "blockstacksystem"
rpc_port = 18332
peer_port = 18333
[connection_options]
read_only_call_limit_write_length = 0
read_only_call_limit_read_length = 100000
read_only_call_limit_write_count = 0
read_only_call_limit_read_count = 30
read_only_call_limit_runtime = 1000000000```
Firewall config
```$ sudo ufw status
Status: active
To Action From
-- ------ ----
80/tcp ALLOW Anywhere
443/tcp ALLOW Anywhere
22 ALLOW Anywhere
127.0.0.1 5432/tcp ALLOW 127.0.0.1
3999 ALLOW Anywhere
20443 ALLOW Anywhere
20444 ALLOW Anywhere
8332 ALLOW Anywhere
8333 ALLOW Anywhere
18333 ALLOW Anywhere
18332 ALLOW Anywhere
80/tcp (v6) ALLOW Anywhere (v6)
443/tcp (v6) ALLOW Anywhere (v6)
22 (v6) ALLOW Anywhere (v6)
3999 (v6) ALLOW Anywhere (v6)
20443 (v6) ALLOW Anywhere (v6)
20444 (v6) ALLOW Anywhere (v6)
8332 (v6) ALLOW Anywhere (v6)
8333 (v6) ALLOW Anywhere (v6)
18333 (v6) ALLOW Anywhere (v6)
18332 (v6) ALLOW Anywhere (v6)```
last log output
docker logs -f stacks-blockchain
```INFO [1660291004.817990] [src/burnchains/burnchain.rs:1306] [main] Syncing Bitcoin blocks: 11.0% (2034650 to 2034651 out of 2316019)
INFO [1660291004.985416] [src/net/http.rs:1602] [p2p] Handle HTTPRequest, verb: GET, peer_addr: 172.18.0.3:54462, path: /v2/info, query:
INFO [1660291005.087021] [src/chainstate/burn/db/sortdb.rs:1930] [chains-coordinator] Reward cycle #33 (2034650): Some((42365dac088b948da07e4fbda5b4211e9c8d1233, 12b98998c11e51864842cbb3e0e5c8f09430d72055747b9c3b767993a1bc1ec3, 46)) reached (F*w), expecting consensus over proof of transfer
INFO [1660291005.087061] [src/chainstate/coordinator/mod.rs:449] [chains-coordinator] Anchor block selected: 12b98998c11e51864842cbb3e0e5c8f09430d72055747b9c3b767993a1bc1ec3
INFO [1660291005.092272] [src/chainstate/burn/db/processing.rs:156] [chains-coordinator] Begin reward-cycle sortition with absent anchor block=Some(12b98998c11e51864842cbb3e0e5c8f09430d72055747b9c3b767993a1bc1ec3)
INFO [1660291005.123244] [src/chainstate/burn/db/sortdb.rs:3726] [chains-coordinator] ACCEPTED(2034651) leader block commit 19e16d77f7d01a33efe4f07442cdae49e4d9f9bd83413ab6ed40086dc0bd4d54 at 2034651,26, apparent_sender: 1HyJaCR6Z3LeEAv9EpvWywiUkwPwe44qJG
INFO [1660291005.123410] [src/chainstate/burn/db/sortdb.rs:3726] [chains-coordinator] ACCEPTED(2034651) leader block commit 9c736f81a484955157d96d2e475aa3bd8112c99a159dba44f4640c8ea5c2a618 at 2034651,33, apparent_sender: 1LER5ScueDhuXYpqsNaPmx7jegCgxzB7dM
INFO [1660291005.157263] [testnet/stacks-node/src/run_loop/neon.rs:654] [main] Downloaded burnchain blocks up to height 2034651; target height is 2034651; next_sortition_height = 2034651, sortition_db_height = 2034650
INFO [1660291005.158703] [testnet/stacks-node/src/neon_node.rs:2148] [main] Received burnchain block #2034651 including block_commit_op (winning) - mzkNNVhtTF9AJfJTawYmbsL4WfoPngk7qv (498f4f3e4de871e290983541b48484fb7873aee3776ea23d24ad2d57b9fd5666)
INFO [1660291005.989526] [src/net/http.rs:1602] [p2p] Handle HTTPRequest, verb: GET, peer_addr: 172.18.0.3:54474, path: /v2/info, query:
INFO [1660291006.802419] [src/net/relay.rs:990] [relayer] Processing newly received Stacks blocks: 6, microblocks: 0, confirmed microblocks: 0
INFO [1660291006.844043] [src/net/relay.rs:990] [relayer] Processing newly received Stacks blocks: 4, microblocks: 0, confirmed microblocks: 0
INFO [1660291006.862786] [src/net/relay.rs:990] [relayer] Processing newly received Stacks blocks: 0, microblocks: 0, confirmed microblocks: 1
INFO [1660291006.995952] [src/net/http.rs:1602] [p2p] Handle HTTPRequest, verb: GET, peer_addr: 172.18.0.3:54488, path: /v2/info, query:
INFO [1660291007.998820] [src/net/http.rs:1602] [p2p] Handle HTTPRequest, verb: GET, peer_addr: 172.18.0.3:54504, path: /v2/info, query:
INFO [1660291009.003617] [src/net/http.rs:1602] [p2p] Handle HTTPRequest, verb: GET, peer_addr: 172.18.0.3:54514, path: /v2/info, query:
INFO [1660291010.006307] [src/net/http.rs:1602] [p2p] Handle HTTPRequest, verb: GET, peer_addr: 172.18.0.3:54530, path: /v2/info, query:
INFO [1660291011.009425] [src/net/http.rs:1602] [p2p] Handle HTTPRequest, verb: GET, peer_addr: 172.18.0.3:56080, path: /v2/info, query:
INFO [1660291012.013040] [src/net/http.rs:1602] [p2p] Handle HTTPRequest, verb: GET, peer_addr: 172.18.0.3:56094, path: /v2/info, query:
INFO [1660291013.016796] [src/net/http.rs:1602] [p2p] Handle HTTPRequest, verb: GET, peer_addr: 172.18.0.3:56096, path: /v2/info, query:
INFO [1660291014.020340] [src/net/http.rs:1602] [p2p] Handle HTTPRequest, verb: GET, peer_addr: 172.18.0.3:56098, path: /v2/info, query:
INFO [1660291015.024865] [src/net/http.rs:1602] [p2p] Handle HTTPRequest, verb: GET, peer_addr: 172.18.0.3:56108, path: /v2/info, query:
INFO [1660291016.028042] [src/net/http.rs:1602] [p2p] Handle HTTPRequest, verb: GET, peer_addr: 172.18.0.3:56120, path: /v2/info, query:
INFO [1660291017.032237] [src/net/http.rs:1602] [p2p] Handle HTTPRequest, verb: GET, peer_addr: 172.18.0.3:56134, path: /v2/info, query:
INFO [1660291018.035117] [src/net/http.rs:1602] [p2p] Handle HTTPRequest, verb: GET, peer_addr: 172.18.0.3:56146, path: /v2/info, query:
INFO [1660291019.039450] [src/net/http.rs:1602] [p2p] Handle HTTPRequest, verb: GET, peer_addr: 172.18.0.3:56148, path: /v2/info, query:
INFO [1660291020.042552] [src/net/http.rs:1602] [p2p] Handle HTTPRequest, verb: GET, peer_addr: 172.18.0.3:56164, path: /v2/info, query:
INFO [1660291021.045902] [src/net/http.rs:1602] [p2p] Handle HTTPRequest, verb: GET, peer_addr: 172.18.0.3:41658, path: /v2/info, query:
INFO [1660291022.080060] [src/net/http.rs:1602] [p2p] Handle HTTPRequest, verb: GET, peer_addr: 172.18.0.3:41662, path: /v2/info, query:
INFO [1660291023.084960] [src/net/http.rs:1602] [p2p] Handle HTTPRequest, verb: GET, peer_addr: 172.18.0.3:41668, path: /v2/info, query:
INFO [1660291024.088691] [src/net/http.rs:1602] [p2p] Handle HTTPRequest, verb: GET, peer_addr: 172.18.0.3:41684, path: /v2/info, query:```
carries on like this.
last log output
docker logs -f stacks-blockchain-api
```{"level":"http","message":"HTTP POST /new_burn_block","req":{"headers":{"content-length":"184","content-type":"application/json","host":"stacks-blockchain-api:3700"},"httpVersion":"1.1","method":"POST","originalUrl":"/new_burn_block","query":{},"url":"/new_burn_block"},"res":{"statusCode":200},"responseTime":3,"timestamp":"2022-08-12T07:56:44.297Z"}
{"level":"http","message":"HTTP POST /new_burn_block","req":{"headers":{"content-length":"184","content-type":"application/json","host":"stacks-blockchain-api:3700"},"httpVersion":"1.1","method":"POST","originalUrl":"/new_burn_block","query":{},"url":"/new_burn_block"},"res":{"statusCode":200},"responseTime":4,"timestamp":"2022-08-12T07:56:44.460Z"}
{"level":"http","message":"HTTP POST /new_burn_block","req":{"headers":{"content-length":"184","content-type":"application/json","host":"stacks-blockchain-api:3700"},"httpVersion":"1.1","method":"POST","originalUrl":"/new_burn_block","query":{},"url":"/new_burn_block"},"res":{"statusCode":200},"responseTime":3,"timestamp":"2022-08-12T07:56:44.605Z"}
{"level":"http","message":"HTTP POST /new_burn_block","req":{"headers":{"content-length":"184","content-type":"application/json","host":"stacks-blockchain-api:3700"},"httpVersion":"1.1","method":"POST","originalUrl":"/new_burn_block","query":{},"url":"/new_burn_block"},"res":{"statusCode":200},"responseTime":3,"timestamp":"2022-08-12T07:56:44.751Z"}
{"level":"http","message":"HTTP POST /new_burn_block","req":{"headers":{"content-length":"184","content-type":"application/json","host":"stacks-blockchain-api:3700"},"httpVersion":"1.1","method":"POST","originalUrl":"/new_burn_block","query":{},"url":"/new_burn_block"},"res":{"statusCode":200},"responseTime":4,"timestamp":"2022-08-12T07:56:45.133Z"}```
----
Update - it seems the api node is unable to connect to postgres db server?
Watching the postgres docker log -40 mins after the database server comes online
2022-08-16 11:29:39.033 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2022-08-16 11:29:39.033 UTC [1] LOG: listening on IPv6 address "::", port 5432
2022-08-16 11:29:39.034 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-08-16 11:29:39.036 UTC [21] LOG: database system was shut down at 2022-08-16 11:26:08 UTC
2022-08-16 11:29:39.038 UTC [1] LOG: database system is ready to accept connections
2022-08-16 12:12:16.021 UTC [93] FATAL: password authentication failed for user "postgres"
2022-08-16 12:12:16.021 UTC [93] DETAIL: Connection matched pg_hba.conf line 100: "host all all all scram-sha-256"
bash: connect: Operation timed out
bash: line 1: /dev/tcp/104.131.82.44/3232: Operation timed out
bash: line 1: echo: write error: Bad file descriptor
bash: line 1: read: read error: 0: Resource temporarily unavailable
cat: read error: Resource temporarily unavailable
bash: connect: Operation timed out
bash: line 1: /dev/tcp/104.131.82.44/3232: Operation timed out
bash: line 1: echo: write error: Bad file descriptor
bash: line 1: read: read error: 0: Resource temporarily unavailable
cat: read error: Resource temporarily unavailable
2022-08-16 12:16:39.108 UTC [1] LOG: received SIGHUP, reloading configuration files
I double checked the password is as set in .env (postgres)
Password for user postgres:
psql (13.7 (Debian 13.7-0+deb11u1), server 14.5)
WARNING: psql major version 13, server major version 14.
Some psql features might not work.
Type "help" for help.
postgres=>
Any help moving this on would be really appreciated as I am hitting Hiro rate limits and need to run an independent full api node?
