cardano-graphql
cardano-graphql copied to clipboard
cardano-graphql schema migrations failing for private testnet
Summary
We have a private testnet up and running, with cardano-node, cardano-ogmios, postgres and cardano-db-sync successfully connected and everything works fine.
Now we want to bring in cardano-graphql but we seem not to get it working. Here is the docker-compose.yml (note that we have to use cardano-db-sync:master otherwise cardano-db-sync would fail to sync with the private testnet which bootstraps into alonzo from block 0):
version: "3.3"
services:
testnet:
build:
context: .
ports:
- "3001:3001"
- "12789:12789"
volumes:
- ${PWD}/shared/configs:/testnet/configs
- ${PWD}/shared/sockets:/testnet/sockets
logging:
driver: "json-file"
options:
max-size: "200k"
max-file: "10"
cardano-ogmios:
command: [
"--host", "0.0.0.0",
"--node-socket", "/shared/sockets/node-pool1.sock",
"--node-config", "/shared/configs/configuration.yaml",
]
image: cardanosolutions/ogmios:v5.2.0
depends_on:
- testnet
logging:
driver: "json-file"
options:
max-size: "400k"
max-file: "20"
ports:
- ${OGMIOS_PORT:-1337}:1337
restart: on-failure
volumes:
- ${PWD}/shared:/shared
postgres:
image: postgres:11.5-alpine
shm_size: 1g
environment:
- POSTGRES_LOGGING=true
- POSTGRES_DB_FILE=/run/secrets/postgres_db
- POSTGRES_PASSWORD_FILE=/run/secrets/postgres_password
- POSTGRES_USER_FILE=/run/secrets/postgres_user
secrets:
- postgres_password
- postgres_user
- postgres_db
ports:
- ${POSTGRES_PORT:-5432}:5432
restart: on-failure
logging:
driver: "json-file"
options:
max-size: "200k"
max-file: "10"
cardano-db-sync:
image: inputoutput/cardano-db-sync:master
command: [
"--config /shared/configs/db-sync-config.yaml",
"--socket-path /shared/sockets/node-pool1.sock"
]
environment:
- EXTENDED=true
- POSTGRES_HOST=postgres
- POSTGRES_USER=postgres
- POSTGRES_PORT=5432
- RESTORE_RECREATE_DB=N
depends_on:
- testnet
- postgres
secrets:
- postgres_password
- postgres_user
- postgres_db
volumes:
- ${PWD}/shared:/shared
# NOTE: uncomment if you want to keep data
# - ${PWD}/dbdata/sync/db-sync-data:/var/lib/cdbsync
restart: on-failure
logging:
driver: "json-file"
options:
max-size: "200k"
max-file: "10"
hasura:
image: inputoutput/cardano-graphql-hasura:master
ports:
- ${HASURA_PORT:-8090}:8080
depends_on:
- "postgres"
restart: on-failure
environment:
- HASURA_GRAPHQL_ENABLE_CONSOLE=true
- HASURA_GRAPHQL_CORS_DOMAIN=http://localhost:9695
secrets:
- postgres_db
- postgres_password
- postgres_user
logging:
driver: "json-file"
options:
max-size: "200k"
max-file: "10"
secrets:
postgres_db:
file: ./db/postgres_db
postgres_password:
file: ./db/postgres_password
postgres_user:
file: ./db/postgres_user
volumes:
db-sync-data:
postgres:
prometheus-data:
When all services are up and running and the socket files of the private testnet nodes are exposed, we are firing up cardano-graphql with another docker-compose:
version: "3.3"
services:
cardano-graphql:
image: inputoutput/cardano-graphql:master
environment:
- CARDANO_NODE_CONFIG_PATH=/shared/configs/configuration-json.json
- CARDANO_NODE_SOCKET_PATH=/shared/sockets/node-pool1.sock
- HASURA_URI=http://hasura:8080
- OGMIOS_HOST=cardano-ogmios
- OGMIOS_PORT=1337
- POSTGRES_HOST=postgres
- POSTGRES_PORT=5432
- POSTGRES_DB_FILE=/run/secrets/postgres_db
- POSTGRES_PASSWORD_FILE=/run/secrets/postgres_password
- POSTGRES_USER_FILE=/run/secrets/postgres_user
- ALLOW_INTROSPECTION=true
- CACHE_ENABLED=true
- LOGGER_MIN_SEVERITY=${LOGGER_MIN_SEVERITY:-debug}
expose:
- ${API_PORT:-3100}
ports:
- ${API_PORT:-3100}:3100
restart: on-failure
secrets:
- postgres_db
- postgres_password
- postgres_user
volumes:
- ${PWD}/shared:/shared
logging:
driver: "json-file"
options:
max-size: "200k"
max-file: "10"
secrets:
postgres_db:
file: ./db/postgres_db
postgres_password:
file: ./db/postgres_password
postgres_user:
file: ./db/postgres_user
volumes:
db-sync-data:
postgres:
prometheus-data:
When cardano-graphql starts we see the following output:
{"name":"cardano-graphql","hostname":"8c1a9ab21496","pid":1,"level":20,"module":"Db","msg":"pgSubscriber: Connected","time":"2022-05-16T17:34:47.695Z","v":0}
{"name":"cardano-graphql","hostname":"8c1a9ab21496","pid":1,"level":30,"module":"Server","msg":"Initializing","time":"2022-05-16T17:34:47.696Z","v":0}
{"name":"cardano-graphql","hostname":"8c1a9ab21496","pid":1,"level":30,"module":"HasuraClient","msg":"Initializing","time":"2022-05-16T17:34:47.811Z","v":0}
{"name":"cardano-graphql","hostname":"8c1a9ab21496","pid":1,"level":20,"module":"HasuraClient","msg":"{\"level\":\"info\",\"msg\":\"Help us improve Hasura! The cli collects anonymized usage stats which\\nallow us to keep improving Hasura at warp speed. To opt-out or read more,\\nvisit https://hasura.io/docs/1.0/graphql/manual/guides/telemetry.html\\n\",\"time\":\"2022-05-16T17:34:47Z\"}\n{\"level\":\"info\",\"msg\":\"Applying migrations...\",\"time\":\"2022-05-16T17:34:50Z\"}\n{\"level\":\"info\",\"msg\":\"nothing to apply\",\"time\":\"2022-05-16T17:34:50Z\"}\n","time":"2022-05-16T17:34:50.114Z","v":0}
{"name":"cardano-graphql","hostname":"8c1a9ab21496","pid":1,"level":20,"module":"HasuraClient","msg":"{\"level\":\"info\",\"msg\":\"Applying migrations...\",\"time\":\"2022-05-16T17:34:50Z\"}\n","time":"2022-05-16T17:34:50.250Z","v":0}
{"name":"cardano-graphql","hostname":"8c1a9ab21496","pid":1,"level":20,"msg":"Applying PostgreSQL schema migrations: Attempt 1 of 10, retrying...","time":"2022-05-16T17:34:50.250Z","v":0}
{"name":"cardano-graphql","hostname":"8c1a9ab21496","pid":1,"level":20,"module":"HasuraClient","msg":"{\"level\":\"info\",\"msg\":\"Applying migrations...\",\"time\":\"2022-05-16T17:34:51Z\"}\n{\"level\":\"info\",\"msg\":\"nothing to apply\",\"time\":\"2022-05-16T17:34:51Z\"}\n","time":"2022-05-16T17:34:51.353Z","v":0}
{"name":"cardano-graphql","hostname":"8c1a9ab21496","pid":1,"level":20,"module":"HasuraClient","msg":"{\"level\":\"info\",\"msg\":\"Applying migrations...\",\"time\":\"2022-05-16T17:34:51Z\"}\n","time":"2022-05-16T17:34:51.488Z","v":0}
This continues until Attempt 10 of 10 followed by a final error output:
{"name":"cardano-graphql","hostname":"8c1a9ab21496","pid":1,"level":50,"msg":"Command failed: /usr/local/bin/hasura --skip-update-check --project /app/packages/api-cardano-db-hasura/hasura/project --endpoint http://hasura:8080 migrate apply --up all\ntime=\"2022-05-16T17:38:16Z\" level=fatal msg=\"apply failed: [postgres-error] query execution failed ($.args[0].args)\\r\\nFile: '1589369664961_init/up.sql'\\r\\n{\\n \\\"check_metadata_consistency\\\": false,
...
FatalError: column pool.reward_addr does not exist\\r\\nHint: Perhaps you meant to reference the column \\\"pool.reward_addr_id\\\".\"\n","time":"2022-05-16T17:38:16.334Z","v":0}
Steps to reproduce the bug
No response
Actual Result
See summary description.
Expected Result
Expected is that cardano-graphql gets up and running without errors and that we can connect to it through http://localhost:3100/graphql.
Environment
Using environment as specified in the docker-compose in the Summary.
cardano-node --version: cardano-node 1.34.0 - linux-x86_64 - ghc-8.10 git rev c23d5d319cd3276575c6ac32458516232e8d2c48
cardano-db-sync --version: cardano-db-sync 12.0.1 - linux-x86_64 - ghc-8.10 git revision 6af53927161a3914598491de6c650da96f3c8c09
Platform
- [X] Linux (Ubuntu)
- [ ] Linux (Other)
- [ ] macOS
- [ ] Windows
Platform version
Ubuntu 22.04
Runtime
- [ ] Node.js
- [ ] Docker
Runtime version
No response
This should work, but I wouldn't expect it to split across docker-compose files given you'd be creating two networks. Why not just add it to the same compose file?
Added to the same compose file, now running into different issue:
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":50,"err":{"errno":-111,"code":"ECONNREFUSED","syscall":"connect","address":"172.18.0.6","port":5432},"msg":"connect ECONNREFUSED 172.18.0.6:5432","time":"2022-07-11T11:16:43.877Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":40,"module":"Db","msg":"pgSubscriber: Reconnecting attempt 1","time":"2022-07-11T11:16:43.879Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":40,"module":"Db","msg":"pgSubscriber: Reconnecting attempt 2","time":"2022-07-11T11:16:44.417Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":40,"module":"Db","msg":"pgSubscriber: Reconnecting attempt 3","time":"2022-07-11T11:16:44.919Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"Db","msg":"pgSubscriber: Connected","time":"2022-07-11T11:16:44.963Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":30,"module":"Server","msg":"Initializing","time":"2022-07-11T11:16:44.963Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":30,"module":"HasuraClient","msg":"Initializing","time":"2022-07-11T11:16:45.088Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"{\"level\":\"info\",\"msg\":\"Help us improve Hasura! The cli collects anonymized usage stats which\\nallow us to keep improving Hasura at warp speed. To opt-out or read more,\\nvisit https://hasura.io/docs/1.0/graphql/manual/guides/telemetry.html\\n\",\"time\":\"2022-07-11T11:16:45Z\"}\n","time":"2022-07-11T11:16:46.344Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Applying PostgreSQL schema migrations: Attempt 1 of 10, retrying...","time":"2022-07-11T11:16:46.344Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"{\"level\":\"info\",\"msg\":\"Applying migrations...\",\"time\":\"2022-07-11T11:16:47Z\"}\n{\"level\":\"info\",\"msg\":\"nothing to apply\",\"time\":\"2022-07-11T11:16:47Z\"}\n","time":"2022-07-11T11:16:47.501Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"{\"level\":\"info\",\"msg\":\"Applying migrations...\",\"time\":\"2022-07-11T11:16:47Z\"}\n{\"level\":\"info\",\"msg\":\"migrations applied\",\"time\":\"2022-07-11T11:16:47Z\"}\n","time":"2022-07-11T11:16:47.720Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"{\"level\":\"info\",\"msg\":\"Clearing metadata...\",\"time\":\"2022-07-11T11:16:47Z\"}\n{\"level\":\"info\",\"msg\":\"Metadata cleared\",\"time\":\"2022-07-11T11:16:47Z\"}\n","time":"2022-07-11T11:16:47.857Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"{\"level\":\"info\",\"msg\":\"Applying metadata...\",\"time\":\"2022-07-11T11:16:47Z\"}\n{\"level\":\"info\",\"msg\":\"Installing plugin cli-ext...\",\"time\":\"2022-07-11T11:16:48Z\"}\n{\"level\":\"info\",\"msg\":\"plugin installed\",\"name\":\"cli-ext\",\"time\":\"2022-07-11T11:16:52Z\"}\n{\"level\":\"info\",\"msg\":\"Metadata applied\",\"time\":\"2022-07-11T11:16:52Z\"}\n","time":"2022-07-11T11:16:52.646Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"graphql-engine setup","time":"2022-07-11T11:16:52.883Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:16:52.888Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 1 of 101, retrying...","time":"2022-07-11T11:16:52.888Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:16:53.894Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 2 of 101, retrying...","time":"2022-07-11T11:16:53.894Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:16:54.950Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 3 of 101, retrying...","time":"2022-07-11T11:16:54.950Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:16:56.057Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 4 of 101, retrying...","time":"2022-07-11T11:16:56.057Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:16:57.220Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 5 of 101, retrying...","time":"2022-07-11T11:16:57.220Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:16:58.441Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 6 of 101, retrying...","time":"2022-07-11T11:16:58.441Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:16:59.723Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 7 of 101, retrying...","time":"2022-07-11T11:16:59.723Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:17:01.066Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 8 of 101, retrying...","time":"2022-07-11T11:17:01.066Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:17:02.479Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 9 of 101, retrying...","time":"2022-07-11T11:17:02.479Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:17:03.962Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 10 of 101, retrying...","time":"2022-07-11T11:17:03.962Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:17:05.518Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 11 of 101, retrying...","time":"2022-07-11T11:17:05.518Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:17:07.151Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 12 of 101, retrying...","time":"2022-07-11T11:17:07.151Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:17:08.866Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 13 of 101, retrying...","time":"2022-07-11T11:17:08.866Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:17:10.666Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 14 of 101, retrying...","time":"2022-07-11T11:17:10.666Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:17:12.556Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 15 of 101, retrying...","time":"2022-07-11T11:17:12.556Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:17:14.540Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 16 of 101, retrying...","time":"2022-07-11T11:17:14.540Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:17:16.625Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"msg":"Detecting DB sync state has reached minimum progress: Attempt 17 of 101, retrying...","time":"2022-07-11T11:17:16.625Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"DB sync state has reached minimum progress","time":"2022-07-11T11:17:18.814Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":20,"module":"HasuraClient","msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:17:18.820Z","v":0}
{"name":"cardano-graphql","hostname":"6f2c6a0b526e","pid":1,"level":50,"msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2022-07-11T11:17:18.820Z","v":0}
It seems that graphql just "gives up" after Attempt 17 of 101, instead of keep trying. The solution was to really start hasura and graphql "later", that is after the testnet has started to produce a few blocks (~50). We start hasura and graphql in a separate docker-compose file, which works just fine.
In this process we also have switched to cardano-node/cli version 1.35 and bumped the versions of the services accordingly:
- cardano-wallet:2022.7.1
- cardano-ogmios:v5.5.2
- postgres:14.4-alpine
- cardano-db-sync:13.0.1
- cardano-graphql-hasura:master (6.2.0 does not work, failing schema migrations)
- cardano-graphql:master (same)
Note that in our private testnet we are bootstrapping into Babbage from epoch 0, as defined in configuration.yml:
TestShelleyHardForkAtEpoch: 0
TestAllegraHardForkAtEpoch: 0
TestMaryHardForkAtEpoch: 0
TestAlonzoHardForkAtEpoch: 0
TestBabbageHardForkAtEpoch: 0