graph-node icon indicating copy to clipboard operation
graph-node copied to clipboard
verified publisher icon graphprotocol

[Bug] Local graph-node in docker shuts immediately after processing event

Open drewstone opened this issue 2 years ago • 4 comments

Bug report

Running a local graph node and I imagine I'm doing something wrong because otherwise this is supremely unreliable.

docker-compose.yml

version: '3'

networks:
  orbit_webb:
    external: true

services:
  graph-node:
    image: graphprotocol/graph-node
    networks:
      - orbit_webb
    ports:
      - '8000:8000'
      - '8001:8001'
      - '8020:8020'
      - '8030:8030'
      - '8040:8040'
    depends_on:
      - ipfs
      - postgres
    environment:
      postgres_host: postgres
      postgres_user: graph-node
      postgres_pass: let-me-in
      postgres_db: graph-node
      ipfs: 'ipfs:5001'
      GRAPH_NODE_CONFIG: '/data/chains-configs/chains.toml'
      GRAPH_LOG: info
    volumes:
      - ./configs:/data/chains-configs

  ipfs:
    image: ipfs/go-ipfs:v0.10.0
    ports:
      - '5001:5001'
    networks:
      - orbit_webb
    volumes:
      - ./data/ipfs:/data/ipfs

  postgres:
    image: postgres
    ports:
      - '5432:5432'
    command: ['postgres', '-cshared_preload_libraries=pg_stat_statements']
    networks:
      - orbit_webb
    environment:
      POSTGRES_USER: graph-node
      POSTGRES_PASSWORD: let-me-in
      POSTGRES_DB: graph-node
      # FIXME: remove this env. var. which we shouldn't need. Introduced by
      # <https://github.com/graphprotocol/graph-node/pull/3511>, maybe as a
      # workaround for https://github.com/docker/for-mac/issues/6270?
      PGDATA: '/var/lib/postgresql/data'
      POSTGRES_INITDB_ARGS: '-E UTF8 --locale=C'
    volumes:
      - ./data/postgres:/var/lib/postgresql/data

Relevant log output

# First instance:

evm-graph-node-1  | Jul 05 00:34:11.403 INFO Graph Node version: 0.31.0 (2023-06-12)
evm-graph-node-1  | Jul 05 00:34:11.413 WARN GRAPH_POI_ACCESS_TOKEN not set; might leak POIs to the public via GraphQL
evm-graph-node-1  | Jul 05 00:34:11.413 INFO Reading configuration file `/data/chains-configs/chains.toml`
evm-graph-node-1  | Jul 05 00:34:11.440 WARN No fork base URL specified, subgraph forking is disabled
evm-graph-node-1  | Jul 05 00:34:11.440 INFO Starting up
evm-graph-node-1  | Jul 05 00:34:11.444 INFO Trying IPFS node at: http://ipfs:5001/
evm-graph-node-1  | Jul 05 00:34:11.546 INFO Creating transport, capabilities: , url: http://athena_chain:8545, provider: athena
evm-graph-node-1  | Jul 05 00:34:11.592 INFO Successfully connected to IPFS node at: http://ipfs:5001/
evm-graph-node-1  | Jul 05 00:34:11.601 INFO Creating transport, capabilities: , url: http://demeter_chain:8545, provider: demeter
evm-graph-node-1  | Jul 05 00:34:11.644 INFO Creating transport, capabilities: , url: http://hermes_chain:8545, provider: hermes
evm-graph-node-1  | Jul 05 00:34:11.705 WARN Expensive queries file not set to a valid file: /etc/graph-node/expensive-queries.txt
evm-graph-node-1  | Jul 05 00:34:12.040 INFO Connecting to Postgres, weight: 1, conn_pool_size: 10, url: postgresql://graph-node:HIDDEN_PASSWORD@postgres:5432/graph-node, pool: main, shard: primary
evm-graph-node-1  | Jul 05 00:34:12.068 INFO Pool successfully connected to Postgres, pool: main, shard: primary, component: Store
evm-graph-node-1  | Jul 05 00:34:12.334 INFO Setting up fdw, pool: main, shard: primary, component: ConnectionPool
evm-graph-node-1  | Jul 05 00:34:12.663 INFO Running migrations, pool: main, shard: primary, component: ConnectionPool
evm-graph-node-1  | Jul 05 00:34:20.781 INFO Migrations finished, pool: main, shard: primary, component: ConnectionPool
evm-graph-node-1  | Jul 05 00:34:20.798 INFO Mapping primary, pool: main, shard: primary, component: ConnectionPool
evm-graph-node-1  | Jul 05 00:34:20.993 INFO Connecting to Ethereum to get network identifier, capabilities: , provider: demeter
evm-graph-node-1  | Jul 05 00:34:21.003 INFO Connecting to Ethereum to get network identifier, capabilities: , provider: hermes
evm-graph-node-1  | Jul 05 00:34:21.006 INFO Connecting to Ethereum to get network identifier, capabilities: , provider: athena
evm-graph-node-1  | Jul 05 00:34:21.028 INFO Connected to Ethereum, capabilities: , network_version: 1688511712377, provider: demeter
evm-graph-node-1  | Jul 05 00:34:21.031 INFO Connected to Ethereum, capabilities: , network_version: 1688511712393, provider: hermes
evm-graph-node-1  | Jul 05 00:34:21.032 INFO Connected to Ethereum, capabilities: , network_version: 1688515985168, provider: athena
evm-graph-node-1  | Jul 05 00:34:21.908 INFO Creating LoadManager in disabled mode, component: LoadManager
evm-graph-node-1  | Jul 05 00:34:21.914 INFO Started block ingestor, network_name: Hermes
evm-graph-node-1  | Jul 05 00:34:21.915 INFO Started block ingestor, network_name: Athena
evm-graph-node-1  | Jul 05 00:34:21.917 INFO Started block ingestor, network_name: Demeter
evm-graph-node-1  | Jul 05 00:34:21.917 INFO Starting firehose block ingestor for network, network_name: Hermes
evm-graph-node-1  | Jul 05 00:34:21.917 INFO Starting firehose block ingestor for network, network_name: Athena
evm-graph-node-1  | Jul 05 00:34:21.917 INFO Starting firehose block ingestor for network, network_name: Demeter
evm-graph-node-1  | Jul 05 00:34:21.926 INFO Starting job runner with 5 jobs, component: JobRunner
evm-graph-node-1  | Jul 05 00:34:21.942 INFO Downloading latest blocks from Ethereum, this may take a few minutes..., provider: hermes, component: EthereumPollingBlockIngestor
evm-graph-node-1  | Jul 05 00:34:21.945 INFO Downloading latest blocks from Ethereum, this may take a few minutes..., provider: athena, component: EthereumPollingBlockIngestor
evm-graph-node-1  | Jul 05 00:34:21.959 INFO Starting JSON-RPC admin server at: http://localhost:8020, component: JsonRpcServer
evm-graph-node-1  | Jul 05 00:34:21.964 INFO Downloading latest blocks from Ethereum, this may take a few minutes..., provider: demeter, component: EthereumPollingBlockIngestor
evm-graph-node-1  | Jul 05 00:34:21.985 INFO Starting GraphQL HTTP server at: http://localhost:8000, component: GraphQLServer
evm-graph-node-1  | Jul 05 00:34:21.991 INFO Starting index node server at: http://localhost:8030, component: IndexNodeServer
evm-graph-node-1  | Jul 05 00:34:21.998 INFO Starting metrics server at: http://localhost:8040, component: MetricsServer
evm-graph-node-1  | Jul 05 00:34:21.998 INFO Starting GraphQL WebSocket server at: ws://localhost:8001, component: SubscriptionServer
evm-graph-node-1  | Jul 05 00:34:22.008 INFO Started all assigned subgraphs, node_id: default, count: 0, component: SubgraphRegistrar
evm-graph-node-1  | Jul 05 00:34:27.915 INFO Received subgraph_create request, params: SubgraphCreateParams { name: SubgraphName("VAnchorAthenaLocal") }, component: JsonRpcServer
evm-graph-node-1  | Jul 05 00:34:33.651 INFO Received subgraph_deploy request, params: SubgraphDeployParams { name: SubgraphName("VAnchorAthenaLocal"), ipfs_hash: DeploymentHash("QmcCEcR1P5kEiBHSM9zCA4F3PsFnVRvUWkS3nEffNCcyf2"), node_id: None, debug_fork: None, history_blocks: None }, component: JsonRpcServer
evm-graph-node-1  | Jul 05 00:34:33.961 INFO Set subgraph start block, block: None, sgd: 0, subgraph_id: QmcCEcR1P5kEiBHSM9zCA4F3PsFnVRvUWkS3nEffNCcyf2, component: SubgraphRegistrar
evm-graph-node-1  | Jul 05 00:34:33.963 INFO Graft base, block: None, base: None, sgd: 0, subgraph_id: QmcCEcR1P5kEiBHSM9zCA4F3PsFnVRvUWkS3nEffNCcyf2, component: SubgraphRegistrar
evm-graph-node-1  | Jul 05 00:34:37.186 INFO Resolve subgraph files using IPFS, n_templates: 0, n_data_sources: 1, sgd: 1, subgraph_id: QmcCEcR1P5kEiBHSM9zCA4F3PsFnVRvUWkS3nEffNCcyf2, component: SubgraphInstanceManager
evm-graph-node-1  | Jul 05 00:34:37.201 INFO Successfully resolved subgraph files using IPFS, features: ø, n_templates: 0, n_data_sources: 1, sgd: 1, subgraph_id: QmcCEcR1P5kEiBHSM9zCA4F3PsFnVRvUWkS3nEffNCcyf2, component: SubgraphInstanceManager
evm-graph-node-1  | Jul 05 00:34:37.356 INFO Starting subgraph writer, queue_size: 5, sgd: 1, subgraph_id: QmcCEcR1P5kEiBHSM9zCA4F3PsFnVRvUWkS3nEffNCcyf2, component: SubgraphInstanceManager
evm-graph-node-1  | Jul 05 00:34:37.474 INFO Data source count at start: 1, sgd: 1, subgraph_id: QmcCEcR1P5kEiBHSM9zCA4F3PsFnVRvUWkS3nEffNCcyf2, component: SubgraphInstanceManager
evm-graph-node-1  | Jul 05 00:34:37.976 INFO Scanning blocks [0, 0], range_size: 1, sgd: 1, subgraph_id: QmcCEcR1P5kEiBHSM9zCA4F3PsFnVRvUWkS3nEffNCcyf2, component: BlockStream
evm-graph-node-1  | Jul 05 00:34:40.000 INFO Done processing trigger, gas_used: 1277282876, data_source: VAnchorTree, handler: handleInsertion, total_ms: 1559, transaction: 0x86fc…0c92, address: 0x91eb…70a3, signature: Insertion(indexed uint256,uint32,uint256,indexed uint256), sgd: 1, subgraph_id: QmcCEcR1P5kEiBHSM9zCA4F3PsFnVRvUWkS3nEffNCcyf2, component: SubgraphInstanceManager
evm-ipfs-1 exited with code 137
evm-graph-node-1  | Jul 05 00:34:40.061 INFO Done processing trigger, gas_used: 146513954, data_source: VAnchorTree, handler: handleInsertion, total_ms: 59, transaction: 0x86fc…0c92, address: 0x91eb…70a3, signature: Insertion(indexed uint256,uint32,uint256,indexed uint256), sgd: 1, subgraph_id: QmcCEcR1P5kEiBHSM9zCA4F3PsFnVRvUWkS3nEffNCcyf2, component: SubgraphInstanceManager
evm-graph-node-1  | Jul 05 00:34:40.113 INFO Done processing trigger, gas_used: 125852206, data_source: VAnchorTree, handler: handleNewCommitment, total_ms: 50, transaction: 0x86fc…0c92, address: 0x91eb…70a3, signature: NewCommitment(uint256,uint256,uint256,bytes), sgd: 1, subgraph_id: QmcCEcR1P5kEiBHSM9zCA4F3PsFnVRvUWkS3nEffNCcyf2, component: SubgraphInstanceManager
evm-graph-node-1  | Jul 05 00:34:40.162 INFO Done processing trigger, gas_used: 125852206, data_source: VAnchorTree, handler: handleNewCommitment, total_ms: 48, transaction: 0x86fc…0c92, address: 0x91eb…70a3, signature: NewCommitment(uint256,uint256,uint256,bytes), sgd: 1, subgraph_id: QmcCEcR1P5kEiBHSM9zCA4F3PsFnVRvUWkS3nEffNCcyf2, component: SubgraphInstanceManager
evm-graph-node-1  | Jul 05 00:34:40.219 INFO Done processing trigger, gas_used: 115348646, data_source: VAnchorTree, handler: handleNewNullifier, total_ms: 56, transaction: 0x86fc…0c92, address: 0x91eb…70a3, signature: NewNullifier(uint256), sgd: 1, subgraph_id: QmcCEcR1P5kEiBHSM9zCA4F3PsFnVRvUWkS3nEffNCcyf2, component: SubgraphInstanceManager
evm-graph-node-1  | Jul 05 00:34:40.281 INFO Done processing trigger, gas_used: 115348646, data_source: VAnchorTree, handler: handleNewNullifier, total_ms: 61, transaction: 0x86fc…0c92, address: 0x91eb…70a3, signature: NewNullifier(uint256), sgd: 1, subgraph_id: QmcCEcR1P5kEiBHSM9zCA4F3PsFnVRvUWkS3nEffNCcyf2, component: SubgraphInstanceManager
evm-graph-node-1  | Jul 05 00:34:40.332 INFO Applying 11 entity operation(s), block_hash: 0x3001bb53af5c25be69e6f923310ecfdb8ad0c95ed84ea757eea3af6b95468f4e, block_number: 40, sgd: 1, subgraph_id: QmcCEcR1P5kEiBHSM9zCA4F3PsFnVRvUWkS3nEffNCcyf2, component: SubgraphInstanceManager
evm-graph-node-1 exited with code 137

Second instance (after restarting the graph-node in docker desktop application)

evm-graph-node-1  | Jul 05 00:35:04.153 INFO Starting subgraph writer, queue_size: 5, sgd: 1, subgraph_id: QmcCEcR1P5kEiBHSM9zCA4F3PsFnVRvUWkS3nEffNCcyf2, component: SubgraphInstanceManager
evm-graph-node-1  | Jul 05 00:35:04.283 INFO Data source count at start: 1, sgd: 1, subgraph_id: QmcCEcR1P5kEiBHSM9zCA4F3PsFnVRvUWkS3nEffNCcyf2, component: SubgraphInstanceManager
evm-graph-node-1  | Jul 05 00:35:04.705 INFO Scanning blocks [0, 0], range_size: 1, sgd: 1, subgraph_id: QmcCEcR1P5kEiBHSM9zCA4F3PsFnVRvUWkS3nEffNCcyf2, component: BlockStream
evm-graph-node-1  | Jul 05 00:35:05.223 INFO Done processing trigger, gas_used: 1277282876, data_source: VAnchorTree, handler: handleInsertion, total_ms: 214, transaction: 0x86fc…0c92, address: 0x91eb…70a3, signature: Insertion(indexed uint256,uint32,uint256,indexed uint256), sgd: 1, subgraph_id: QmcCEcR1P5kEiBHSM9zCA4F3PsFnVRvUWkS3nEffNCcyf2, component: SubgraphInstanceManager
evm-graph-node-1  | Jul 05 00:35:05.363 INFO Done processing trigger, gas_used: 146513954, data_source: VAnchorTree, handler: handleInsertion, total_ms: 138, transaction: 0x86fc…0c92, address: 0x91eb…70a3, signature: Insertion(indexed uint256,uint32,uint256,indexed uint256), sgd: 1, subgraph_id: QmcCEcR1P5kEiBHSM9zCA4F3PsFnVRvUWkS3nEffNCcyf2, component: SubgraphInstanceManager
evm-graph-node-1 exited with code 137



### IPFS hash

_No response_

### Subgraph name or link to explorer

_No response_

### Some information to help us out

- [ ] Tick this box if this bug is caused by a regression found in the latest release.
- [ ] Tick this box if this bug is specific to the hosted service.
- [X] I have searched the issue tracker to make sure this issue is not a duplicate.

### OS information

macOS

drewstone avatar Jul 05 '23 00:07 drewstone

Code 137 could indicate OOM kill. You could check that you have enough memory for the graph to run on your machine. You could run top/htop during graph execution.

SozinM avatar Jul 05 '23 15:07 SozinM

@drewstone are you on an M1 Mac? Are you running an up to date version of Docker Desktop?

azf20 avatar Jul 10 '23 10:07 azf20

Yes I am on M1. I think it’s up to date but will update today.

On Mon, Jul 10, 2023 at 6:36 AM Adam Fuller @.***> wrote:

@drewstone https://github.com/drewstone are you on an M1 Mac? Are you running an up to date version of Docker Desktop?

— Reply to this email directly, view it on GitHub https://github.com/graphprotocol/graph-node/issues/4740#issuecomment-1628680248, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADELLF5WHJD6LWAOTMKKDS3XPPLLXANCNFSM6AAAAAAZ6IENY4 . You are receiving this because you were mentioned.Message ID: @.***>

drewstone avatar Jul 10 '23 12:07 drewstone

Looks like this issue has been open for 6 months with no activity. Is it still relevant? If not, please remember to close it.

github-actions[bot] avatar Jan 09 '24 00:01 github-actions[bot]