mempool
mempool copied to clipboard
Published
20 hours ago •
mempool
mempool
Mempool backend memory usage problems
Description
I'm running the mempool backend image in a kubernetes pod. The backend does not use all the available memory of the pod, and instead always lists the memory usage as out of 2.05Gb. I have increased the pod memory request to 8Gb and it still won't go beyond 2.05Gb. Additionally, the pod restarts after a while due to an out of memory error after syncing up to about 40%. I have noticed that the percentage upto which it syncs before crashing does increase as I increase the pod's memory request, so perhaps it reporting the memory usage out of 2.05Gb is incorrect.
Version
v3.0.0-dev1
Steps to reproduce
Expected behaviour
Mempool backend should use the memory available on the pod.
Actual behaviour
Mempool backend logs show memory usage out of 2.05Gb even when the memory for the pod is set much higher.
Screenshots
Device or machine
Additional info
Nov 9 14:30:45 [116] NOTICE: Starting Mempool Server... (79fb0a3)
Nov 9 14:30:46 [116] DEBUG: pools-v2.json sha | Current: null | Github: 300b2f4b0fefbeb87a452ef972df20c789048649
Nov 9 14:30:46 [116] INFO: [Mining] Downloading pools-v2.json for the first time from https://raw.githubusercontent.com/mempool/mining-pools/master/pools-v2.json over clearnet
Nov 9 14:30:46 [116] INFO: Mining pools-v2.json (300b2f4b0fefbeb87a452ef972df20c789048649) import completed (no database)
Nov 9 14:30:49 [116] DEBUG: Updated orphaned blocks cache. Found 13 orphaned blocks
Nov 9 14:30:49 [116] NOTICE: Mempool Server is running on port 8999
Nov 9 14:30:51 [116] DEBUG: Initial difficulty adjustment data set.
Nov 9 14:30:51 [116] DEBUG: New block found (#815990)!
Nov 9 14:30:52 [116] DEBUG: Memory usage: 0.04 GB / 2.05 GB
Nov 9 14:30:55 [116] DEBUG: Updated orphaned blocks cache. Found 13 orphaned blocks
Nov 9 14:30:56 [116] DEBUG: Indexing tx 1 of 3713 in block #815990
Nov 9 14:30:59 [116] DEBUG: Indexing tx 372 of 3713 in block #815990
Nov 9 14:31:00 [116] DEBUG: Indexing tx 743 of 3713 in block #815990
Nov 9 14:31:01 [116] DEBUG: Indexing tx 1114 of 3713 in block #815990
Nov 9 14:31:02 [116] DEBUG: Indexing tx 1485 of 3713 in block #815990
Nov 9 14:31:03 [116] DEBUG: Indexing tx 1856 of 3713 in block #815990
Nov 9 14:31:04 [116] DEBUG: Indexing tx 2227 of 3713 in block #815990
Nov 9 14:31:05 [116] DEBUG: Indexing tx 2598 of 3713 in block #815990
Nov 9 14:31:07 [116] DEBUG: Indexing tx 2969 of 3713 in block #815990
Nov 9 14:31:08 [116] DEBUG: Indexing tx 3340 of 3713 in block #815990
Nov 9 14:31:09 [116] DEBUG: Indexing tx 3711 of 3713 in block #815990
Nov 9 14:31:09 [116] DEBUG: Indexing tx 3713 of 3713 in block #815990
Nov 9 14:31:09 [116] DEBUG: 0 of 3713 found in mempool. 3713 fetched through backend service.
Nov 9 14:31:09 [116] DEBUG: 0 websocket clients | 0 connected | 0 disconnected | (+0)
Nov 9 14:31:09 [116] DEBUG: Mempool blocks calculated in 0 seconds
Nov 9 14:31:09 [116] DEBUG: New block found (#815991)!
Nov 9 14:31:12 [116] DEBUG: Updated orphaned blocks cache. Found 13 orphaned blocks
Nov 9 14:31:13 [116] DEBUG: Indexing tx 1 of 3735 in block #815991
Nov 9 14:31:15 [116] DEBUG: Indexing tx 375 of 3735 in block #815991
Nov 9 14:31:16 [116] DEBUG: Indexing tx 749 of 3735 in block #815991
Nov 9 14:31:17 [116] DEBUG: Indexing tx 1123 of 3735 in block #815991
Nov 9 14:31:18 [116] DEBUG: Indexing tx 1497 of 3735 in block #815991
Nov 9 14:31:20 [116] DEBUG: Indexing tx 1871 of 3735 in block #815991
Nov 9 14:31:21 [116] DEBUG: Indexing tx 2245 of 3735 in block #815991
Nov 9 14:31:22 [116] DEBUG: Indexing tx 2619 of 3735 in block #815991
Nov 9 14:31:23 [116] DEBUG: Indexing tx 2993 of 3735 in block #815991
Nov 9 14:31:24 [116] DEBUG: Indexing tx 3367 of 3735 in block #815991
Nov 9 14:31:25 [116] DEBUG: Indexing tx 3735 of 3735 in block #815991
Nov 9 14:31:25 [116] DEBUG: 0 of 3735 found in mempool. 3735 fetched through backend service.
Nov 9 14:31:25 [116] DEBUG: 0 websocket clients | 0 connected | 0 disconnected | (+0)
Nov 9 14:31:25 [116] DEBUG: Mempool blocks calculated in 0 seconds
Nov 9 14:31:25 [116] DEBUG: New block found (#815992)!
Nov 9 14:31:29 [116] DEBUG: Updated orphaned blocks cache. Found 13 orphaned blocks
Nov 9 14:31:31 [116] DEBUG: Indexing tx 1 of 3410 in block #815992
Nov 9 14:31:32 [116] DEBUG: Indexing tx 342 of 3410 in block #815992
Nov 9 14:31:34 [116] DEBUG: Indexing tx 683 of 3410 in block #815992
Nov 9 14:31:35 [116] DEBUG: Indexing tx 1024 of 3410 in block #815992
Nov 9 14:31:36 [116] DEBUG: Indexing tx 1365 of 3410 in block #815992
Nov 9 14:31:37 [116] DEBUG: Indexing tx 1706 of 3410 in block #815992
Nov 9 14:31:38 [116] DEBUG: Indexing tx 2047 of 3410 in block #815992
Nov 9 14:31:39 [116] DEBUG: Indexing tx 2388 of 3410 in block #815992
Nov 9 14:31:40 [116] DEBUG: Indexing tx 2729 of 3410 in block #815992
Nov 9 14:31:42 [116] DEBUG: Indexing tx 3070 of 3410 in block #815992
Nov 9 14:31:43 [116] DEBUG: Indexing tx 3410 of 3410 in block #815992
Nov 9 14:31:43 [116] DEBUG: 0 of 3410 found in mempool. 3410 fetched through backend service.
Nov 9 14:31:43 [116] DEBUG: 0 websocket clients | 0 connected | 0 disconnected | (+0)
Nov 9 14:31:43 [116] DEBUG: Mempool blocks calculated in 0 seconds
Nov 9 14:31:43 [116] DEBUG: New block found (#815993)!
Nov 9 14:31:46 [116] DEBUG: Updated orphaned blocks cache. Found 13 orphaned blocks
Nov 9 14:31:49 [116] DEBUG: Indexing tx 1 of 3539 in block #815993
Nov 9 14:31:50 [116] DEBUG: Indexing tx 355 of 3539 in block #815993
Nov 9 14:31:51 [116] DEBUG: Indexing tx 709 of 3539 in block #815993
Nov 9 14:31:52 [116] DEBUG: Memory usage: 0.09 GB / 2.05 GB
Nov 9 14:31:53 [116] DEBUG: Indexing tx 1063 of 3539 in block #815993
Nov 9 14:31:54 [116] DEBUG: Indexing tx 1417 of 3539 in block #815993
Nov 9 14:31:55 [116] DEBUG: Indexing tx 1771 of 3539 in block #815993
Nov 9 14:31:56 [116] DEBUG: Indexing tx 2125 of 3539 in block #815993
Nov 9 14:31:57 [116] DEBUG: Indexing tx 2479 of 3539 in block #815993
Nov 9 14:31:58 [116] DEBUG: Indexing tx 2833 of 3539 in block #815993
Nov 9 14:32:00 [116] DEBUG: Indexing tx 3187 of 3539 in block #815993
Nov 9 14:32:01 [116] DEBUG: Indexing tx 3539 of 3539 in block #815993
Nov 9 14:32:01 [116] DEBUG: 0 of 3539 found in mempool. 3539 fetched through backend service.
Nov 9 14:32:01 [116] DEBUG: 0 websocket clients | 0 connected | 0 disconnected | (+0)
Nov 9 14:32:01 [116] DEBUG: Mempool blocks calculated in 0 seconds
Nov 9 14:32:01 [116] DEBUG: New block found (#815994)!
Nov 9 14:32:04 [116] DEBUG: Updated orphaned blocks cache. Found 13 orphaned blocks
Nov 9 14:32:07 [116] DEBUG: Indexing tx 1 of 3422 in block #815994
Nov 9 14:32:07 [116] DEBUG: Indexing tx 343 of 3422 in block #815994
Nov 9 14:32:08 [116] DEBUG: Indexing tx 685 of 3422 in block #815994
Nov 9 14:32:10 [116] DEBUG: Indexing tx 1027 of 3422 in block #815994
Nov 9 14:32:11 [116] DEBUG: Indexing tx 1369 of 3422 in block #815994
Nov 9 14:32:11 [116] DEBUG: Indexing tx 1711 of 3422 in block #815994
Nov 9 14:32:12 [116] DEBUG: Indexing tx 2053 of 3422 in block #815994
Nov 9 14:32:14 [116] DEBUG: Indexing tx 2395 of 3422 in block #815994
Nov 9 14:32:15 [116] DEBUG: Indexing tx 2737 of 3422 in block #815994
Nov 9 14:32:16 [116] DEBUG: Indexing tx 3079 of 3422 in block #815994
Nov 9 14:32:17 [116] DEBUG: Indexing tx 3421 of 3422 in block #815994
Nov 9 14:32:17 [116] DEBUG: Indexing tx 3422 of 3422 in block #815994
Nov 9 14:32:17 [116] DEBUG: 0 of 3422 found in mempool. 3422 fetched through backend service.
Nov 9 14:32:17 [116] DEBUG: 0 websocket clients | 0 connected | 0 disconnected | (+0)
Nov 9 14:32:17 [116] DEBUG: Mempool blocks calculated in 0 seconds
Nov 9 14:32:17 [116] DEBUG: New block found (#815995)!
Nov 9 14:32:20 [116] DEBUG: Updated orphaned blocks cache. Found 13 orphaned blocks
Nov 9 14:32:21 [116] DEBUG: Indexing tx 1 of 2525 in block #815995
Nov 9 14:32:22 [116] DEBUG: Indexing tx 254 of 2525 in block #815995
Nov 9 14:32:23 [116] DEBUG: Indexing tx 507 of 2525 in block #815995
Nov 9 14:32:23 [116] DEBUG: Indexing tx 760 of 2525 in block #815995
Nov 9 14:32:24 [116] DEBUG: Indexing tx 1013 of 2525 in block #815995
Nov 9 14:32:25 [116] DEBUG: Indexing tx 1266 of 2525 in block #815995
Nov 9 14:32:26 [116] DEBUG: Indexing tx 1519 of 2525 in block #815995
Nov 9 14:32:27 [116] DEBUG: Indexing tx 1772 of 2525 in block #815995
Nov 9 14:32:28 [116] DEBUG: Indexing tx 2025 of 2525 in block #815995
Nov 9 14:32:29 [116] DEBUG: Indexing tx 2278 of 2525 in block #815995
Nov 9 14:32:30 [116] DEBUG: Indexing tx 2525 of 2525 in block #815995
Nov 9 14:32:30 [116] DEBUG: 0 of 2525 found in mempool. 2525 fetched through backend service.
Nov 9 14:32:30 [116] DEBUG: 0 websocket clients | 0 connected | 0 disconnected | (+0)
Nov 9 14:32:30 [116] DEBUG: Mempool blocks calculated in 0 seconds
Nov 9 14:32:30 [116] DEBUG: New block found (#815996)!
Nov 9 14:32:34 [116] DEBUG: Updated orphaned blocks cache. Found 13 orphaned blocks
Nov 9 14:32:36 [116] DEBUG: Indexing tx 1 of 2967 in block #815996
Nov 9 14:32:37 [116] DEBUG: Indexing tx 298 of 2967 in block #815996
Nov 9 14:32:37 [116] DEBUG: Indexing tx 595 of 2967 in block #815996
Nov 9 14:32:38 [116] DEBUG: Indexing tx 892 of 2967 in block #815996
Nov 9 14:32:40 [116] DEBUG: Indexing tx 1189 of 2967 in block #815996
Nov 9 14:32:41 [116] DEBUG: Indexing tx 1486 of 2967 in block #815996
Nov 9 14:32:42 [116] DEBUG: Indexing tx 1783 of 2967 in block #815996
Nov 9 14:32:42 [116] DEBUG: Indexing tx 2080 of 2967 in block #815996
Nov 9 14:32:43 [116] DEBUG: Indexing tx 2377 of 2967 in block #815996
Nov 9 14:32:45 [116] DEBUG: Indexing tx 2674 of 2967 in block #815996
Nov 9 14:32:46 [116] DEBUG: Indexing tx 2967 of 2967 in block #815996
Nov 9 14:32:46 [116] DEBUG: 0 of 2967 found in mempool. 2967 fetched through backend service.
Nov 9 14:32:46 [116] DEBUG: 0 websocket clients | 0 connected | 0 disconnected | (+0)
Nov 9 14:32:46 [116] DEBUG: Mempool blocks calculated in 0 seconds
Nov 9 14:32:46 [116] DEBUG: New block found (#815997)!
Nov 9 14:32:48 [116] DEBUG: Updated orphaned blocks cache. Found 13 orphaned blocks
Nov 9 14:32:51 [116] ERR: $updateBlocks stalled at "getting block data for 815997"
Nov 9 14:32:51 [116] DEBUG: Indexing tx 1 of 3445 in block #815997
Nov 9 14:32:52 [116] DEBUG: Memory usage: 0.10 GB / 2.05 GB
Nov 9 14:32:52 [116] DEBUG: Indexing tx 346 of 3445 in block #815997
Nov 9 14:32:54 [116] DEBUG: Indexing tx 691 of 3445 in block #815997
Nov 9 14:32:55 [116] DEBUG: Indexing tx 1036 of 3445 in block #815997
Nov 9 14:32:56 [116] DEBUG: Indexing tx 1381 of 3445 in block #815997
Nov 9 14:32:57 [116] DEBUG: Indexing tx 1726 of 3445 in block #815997
Nov 9 14:32:59 [116] DEBUG: Indexing tx 2071 of 3445 in block #815997
Nov 9 14:33:00 [116] DEBUG: Indexing tx 2416 of 3445 in block #815997
Nov 9 14:33:01 [116] DEBUG: Indexing tx 2761 of 3445 in block #815997
Nov 9 14:33:02 [116] DEBUG: Indexing tx 3106 of 3445 in block #815997
Nov 9 14:33:06 [116] DEBUG: Indexing tx 3445 of 3445 in block #815997
Nov 9 14:33:06 [116] DEBUG: 0 of 3445 found in mempool. 3445 fetched through backend service.
Nov 9 14:33:06 [116] DEBUG: 0 websocket clients | 0 connected | 0 disconnected | (+0)
Nov 9 14:33:06 [116] DEBUG: Mempool blocks calculated in 0 seconds
Nov 9 14:33:08 [116] DEBUG: New block found (#815998)!
Nov 9 14:33:11 [116] DEBUG: Updated orphaned blocks cache. Found 13 orphaned blocks
Nov 9 14:33:13 [116] DEBUG: Indexing tx 1 of 4548 in block #815998
Nov 9 14:33:25 [116] DEBUG: Indexing tx 456 of 4548 in block #815998
Nov 9 14:33:27 [116] DEBUG: Indexing tx 911 of 4548 in block #815998
Nov 9 14:33:29 [116] DEBUG: Indexing tx 1366 of 4548 in block #815998
Nov 9 14:33:31 [116] DEBUG: Indexing tx 1821 of 4548 in block #815998
Nov 9 14:33:32 [116] DEBUG: Indexing tx 2276 of 4548 in block #815998
Nov 9 14:33:34 [116] DEBUG: Indexing tx 2731 of 4548 in block #815998
Nov 9 14:33:36 [116] DEBUG: Indexing tx 3186 of 4548 in block #815998
Nov 9 14:33:37 [116] DEBUG: Indexing tx 3641 of 4548 in block #815998
Nov 9 14:33:39 [116] DEBUG: Indexing tx 4096 of 4548 in block #815998
Nov 9 14:33:41 [116] DEBUG: Indexing tx 4548 of 4548 in block #815998
Nov 9 14:33:41 [116] DEBUG: 0 of 4548 found in mempool. 4548 fetched through backend service.
Nov 9 14:33:41 [116] DEBUG: 0 websocket clients | 0 connected | 0 disconnected | (+0)
Nov 9 14:33:41 [116] DEBUG: Mempool blocks calculated in 0 seconds
Nov 9 14:33:43 [116] DEBUG: New block found (#815999)!
Nov 9 14:33:46 [116] DEBUG: Updated orphaned blocks cache. Found 13 orphaned blocks
Nov 9 14:33:48 [116] DEBUG: Indexing tx 1 of 4156 in block #815999
Nov 9 14:33:50 [116] DEBUG: Indexing tx 417 of 4156 in block #815999
Nov 9 14:33:51 [116] DEBUG: Indexing tx 833 of 4156 in block #815999
Nov 9 14:33:52 [116] DEBUG: Memory usage: 0.10 GB / 2.05 GB
Nov 9 14:33:53 [116] DEBUG: Indexing tx 1249 of 4156 in block #815999
Nov 9 14:33:54 [116] DEBUG: Indexing tx 1665 of 4156 in block #815999
Nov 9 14:33:55 [116] DEBUG: Indexing tx 2081 of 4156 in block #815999
Nov 9 14:33:58 [116] DEBUG: Indexing tx 2497 of 4156 in block #815999
Nov 9 14:34:00 [116] DEBUG: Indexing tx 2913 of 4156 in block #815999
Nov 9 14:34:01 [116] DEBUG: Indexing tx 3329 of 4156 in block #815999
Nov 9 14:34:03 [116] DEBUG: Indexing tx 3745 of 4156 in block #815999
Nov 9 14:34:05 [116] DEBUG: Indexing tx 4156 of 4156 in block #815999
Nov 9 14:34:05 [116] DEBUG: 0 of 4156 found in mempool. 4156 fetched through backend service.
Nov 9 14:34:05 [116] DEBUG: 0 websocket clients | 0 connected | 0 disconnected | (+0)
Nov 9 14:34:05 [116] DEBUG: Mempool blocks calculated in 0 seconds
Nov 9 14:34:07 [116] DEBUG: Updating mempool...
Nov 9 14:34:52 [116] DEBUG: Memory usage: 0.59 GB / 2.05 GB
Nov 9 14:35:49 [116] DEBUG: fetched 9868 transactions
Nov 9 14:35:49 [116] DEBUG: Error finding 132 transactions in the mempool:
Nov 9 14:35:49 [116] DEBUG: Mempool is synchronizing. Processed 9868/180748 txs (5%)
Nov 9 14:35:53 [116] DEBUG: Memory usage: 0.40 GB / 2.05 GB
Nov 9 14:36:05 [116] DEBUG: fetched 9995 transactions
Nov 9 14:36:05 [116] DEBUG: Error finding 5 transactions in the mempool:
Nov 9 14:36:05 [116] DEBUG: Mempool is synchronizing. Processed 19863/180748 txs (11%)
Nov 9 14:36:07 [116] ERR: $updateMempool stalled at "fetched new transactions"
Nov 9 14:36:26 [116] DEBUG: fetched 9996 transactions
Nov 9 14:36:26 [116] DEBUG: Error finding 4 transactions in the mempool:
Nov 9 14:36:26 [116] DEBUG: Mempool is synchronizing. Processed 29859/180748 txs (17%)
Nov 9 14:36:44 [116] DEBUG: fetched 10000 transactions
Nov 9 14:36:44 [116] DEBUG: Mempool is synchronizing. Processed 39859/180748 txs (22%)
Nov 9 14:36:53 [116] DEBUG: Memory usage: 0.31 GB / 2.05 GB
Nov 9 14:37:03 [116] DEBUG: fetched 9997 transactions
Nov 9 14:37:03 [116] DEBUG: Error finding 3 transactions in the mempool:
Nov 9 14:37:03 [116] DEBUG: Mempool is synchronizing. Processed 49856/180748 txs (28%)
Nov 9 14:37:31 [116] DEBUG: fetched 9999 transactions
Nov 9 14:37:31 [116] DEBUG: Error finding 1 transactions in the mempool:
Nov 9 14:37:31 [116] DEBUG: Mempool is synchronizing. Processed 59855/180748 txs (33%)
Nov 9 14:37:49 [116] DEBUG: fetched 10000 transactions
Nov 9 14:37:49 [116] DEBUG: Mempool is synchronizing. Processed 69855/180748 txs (39%)
Nov 9 14:37:53 [116] DEBUG: Memory usage: 0.51 GB / 2.05 GB
I found that setting NODE_OPTIONS: --max-old-space-size=6500 as an environment variable on the pod sets the memory that nodejs is allowed to use.
Even with that amount of memory, the pod keeps crashing after reaching 65% sync with a OOM error with exit code 137.
Nov 10 11:05:18 [116] NOTICE: Starting Mempool Server... (79fb0a3)
Nov 10 11:05:18 [116] DEBUG: pools-v2.json sha | Current: null | Github: 300b2f4b0fefbeb87a452ef972df20c789048649
Nov 10 11:05:18 [116] INFO: [Mining] Downloading pools-v2.json for the first time from https://raw.githubusercontent.com/mempool/mining-pools/master/pools-v2.json over clearnet
Nov 10 11:05:18 [116] INFO: Mining pools-v2.json (300b2f4b0fefbeb87a452ef972df20c789048649) import completed (no database)
Nov 10 11:05:20 [116] DEBUG: Updated orphaned blocks cache. Found 13 orphaned blocks
Nov 10 11:05:20 [116] NOTICE: Mempool Server is running on port 8999
Nov 10 11:05:22 [116] DEBUG: Initial difficulty adjustment data set.
Nov 10 11:05:22 [116] DEBUG: New block found (#816128)!
Nov 10 11:05:23 [116] DEBUG: Memory usage: 0.05 GB / 6.39 GB
Nov 10 11:05:24 [116] DEBUG: Updated orphaned blocks cache. Found 13 orphaned blocks
Nov 10 11:05:26 [116] DEBUG: Indexing tx 1 of 4267 in block #816128
Nov 10 11:05:28 [116] DEBUG: Indexing tx 428 of 4267 in block #816128
Nov 10 11:05:29 [116] DEBUG: Indexing tx 855 of 4267 in block #816128
Nov 10 11:05:31 [116] DEBUG: Indexing tx 1282 of 4267 in block #816128
Nov 10 11:05:33 [116] DEBUG: Indexing tx 1709 of 4267 in block #816128
Nov 10 11:05:35 [116] DEBUG: Indexing tx 2136 of 4267 in block #816128
Nov 10 11:05:36 [116] DEBUG: Indexing tx 2563 of 4267 in block #816128
Nov 10 11:05:39 [116] DEBUG: Indexing tx 2990 of 4267 in block #816128
Nov 10 11:05:40 [116] DEBUG: Indexing tx 3417 of 4267 in block #816128
Nov 10 11:05:41 [116] DEBUG: Indexing tx 3844 of 4267 in block #816128
Nov 10 11:05:43 [116] DEBUG: Indexing tx 4267 of 4267 in block #816128
Nov 10 11:05:43 [116] DEBUG: 0 of 4267 found in mempool. 4267 fetched through backend service.
Nov 10 11:05:43 [116] DEBUG: 0 websocket clients | 0 connected | 0 disconnected | (+0)
Nov 10 11:05:43 [116] DEBUG: Mempool blocks calculated in 0 seconds
Nov 10 11:05:43 [116] DEBUG: New block found (#816129)!
Nov 10 11:05:47 [116] DEBUG: Updated orphaned blocks cache. Found 13 orphaned blocks
Nov 10 11:05:48 [116] DEBUG: Indexing tx 1 of 4207 in block #816129
Nov 10 11:05:50 [116] DEBUG: Indexing tx 422 of 4207 in block #816129
Nov 10 11:05:52 [116] DEBUG: Indexing tx 843 of 4207 in block #816129
Nov 10 11:05:53 [116] DEBUG: Indexing tx 1264 of 4207 in block #816129
Nov 10 11:05:55 [116] DEBUG: Indexing tx 1685 of 4207 in block #816129
Nov 10 11:05:56 [116] DEBUG: Indexing tx 2106 of 4207 in block #816129
Nov 10 11:05:58 [116] DEBUG: Indexing tx 2527 of 4207 in block #816129
Nov 10 11:05:59 [116] DEBUG: Indexing tx 2948 of 4207 in block #816129
Nov 10 11:06:00 [116] DEBUG: Indexing tx 3369 of 4207 in block #816129
Nov 10 11:06:02 [116] DEBUG: Indexing tx 3790 of 4207 in block #816129
Nov 10 11:06:03 [116] DEBUG: Indexing tx 4207 of 4207 in block #816129
Nov 10 11:06:03 [116] DEBUG: 0 of 4207 found in mempool. 4207 fetched through backend service.
Nov 10 11:06:03 [116] DEBUG: 0 websocket clients | 0 connected | 0 disconnected | (+0)
Nov 10 11:06:03 [116] DEBUG: Mempool blocks calculated in 0 seconds
Nov 10 11:06:03 [116] DEBUG: New block found (#816130)!
Nov 10 11:06:06 [116] DEBUG: Updated orphaned blocks cache. Found 13 orphaned blocks
Nov 10 11:06:08 [116] DEBUG: Indexing tx 1 of 4480 in block #816130
Nov 10 11:06:09 [116] DEBUG: Indexing tx 449 of 4480 in block #816130
Nov 10 11:06:10 [116] DEBUG: Indexing tx 897 of 4480 in block #816130
Nov 10 11:06:12 [116] DEBUG: Indexing tx 1345 of 4480 in block #816130
Nov 10 11:06:13 [116] DEBUG: Indexing tx 1793 of 4480 in block #816130
Nov 10 11:06:14 [116] DEBUG: Indexing tx 2241 of 4480 in block #816130
Nov 10 11:06:16 [116] DEBUG: Indexing tx 2689 of 4480 in block #816130
Nov 10 11:06:17 [116] DEBUG: Indexing tx 3137 of 4480 in block #816130
Nov 10 11:06:18 [116] DEBUG: Indexing tx 3585 of 4480 in block #816130
Nov 10 11:06:19 [116] DEBUG: Indexing tx 4033 of 4480 in block #816130
Nov 10 11:06:20 [116] DEBUG: Indexing tx 4480 of 4480 in block #816130
Nov 10 11:06:20 [116] DEBUG: 0 of 4480 found in mempool. 4480 fetched through backend service.
Nov 10 11:06:21 [116] DEBUG: 0 websocket clients | 0 connected | 0 disconnected | (+0)
Nov 10 11:06:21 [116] DEBUG: Mempool blocks calculated in 0 seconds
Nov 10 11:06:21 [116] DEBUG: New block found (#816131)!
Nov 10 11:06:23 [116] DEBUG: Memory usage: 0.10 GB / 6.39 GB
Nov 10 11:06:23 [116] DEBUG: Updated orphaned blocks cache. Found 13 orphaned blocks
Nov 10 11:06:25 [116] DEBUG: Indexing tx 1 of 4203 in block #816131
Nov 10 11:06:28 [116] DEBUG: Indexing tx 421 of 4203 in block #816131
Nov 10 11:06:29 [116] DEBUG: Indexing tx 841 of 4203 in block #816131
Nov 10 11:06:30 [116] DEBUG: Indexing tx 1261 of 4203 in block #816131
Nov 10 11:06:32 [116] DEBUG: Indexing tx 1681 of 4203 in block #816131
Nov 10 11:06:33 [116] DEBUG: Indexing tx 2101 of 4203 in block #816131
Nov 10 11:06:35 [116] DEBUG: Indexing tx 2521 of 4203 in block #816131
Nov 10 11:06:36 [116] DEBUG: Indexing tx 2941 of 4203 in block #816131
Nov 10 11:06:37 [116] DEBUG: Indexing tx 3361 of 4203 in block #816131
Nov 10 11:06:39 [116] DEBUG: Indexing tx 3781 of 4203 in block #816131
Nov 10 11:06:40 [116] DEBUG: Indexing tx 4201 of 4203 in block #816131
Nov 10 11:06:40 [116] DEBUG: Indexing tx 4203 of 4203 in block #816131
Nov 10 11:06:40 [116] DEBUG: 0 of 4203 found in mempool. 4203 fetched through backend service.
Nov 10 11:06:40 [116] DEBUG: 0 websocket clients | 0 connected | 0 disconnected | (+0)
Nov 10 11:06:40 [116] DEBUG: Mempool blocks calculated in 0 seconds
Nov 10 11:06:40 [116] DEBUG: New block found (#816132)!
Nov 10 11:06:43 [116] DEBUG: Updated orphaned blocks cache. Found 13 orphaned blocks
Nov 10 11:06:45 [116] DEBUG: Indexing tx 1 of 4115 in block #816132
Nov 10 11:06:47 [116] DEBUG: Indexing tx 413 of 4115 in block #816132
Nov 10 11:06:48 [116] DEBUG: Indexing tx 825 of 4115 in block #816132
Nov 10 11:06:49 [116] DEBUG: Indexing tx 1237 of 4115 in block #816132
Nov 10 11:06:51 [116] DEBUG: Indexing tx 1649 of 4115 in block #816132
Nov 10 11:06:52 [116] DEBUG: Indexing tx 2061 of 4115 in block #816132
Nov 10 11:06:53 [116] DEBUG: Indexing tx 2473 of 4115 in block #816132
Nov 10 11:06:54 [116] DEBUG: Indexing tx 2885 of 4115 in block #816132
Nov 10 11:06:56 [116] DEBUG: Indexing tx 3297 of 4115 in block #816132
Nov 10 11:06:57 [116] DEBUG: Indexing tx 3709 of 4115 in block #816132
Nov 10 11:06:58 [116] DEBUG: Indexing tx 4115 of 4115 in block #816132
Nov 10 11:06:58 [116] DEBUG: 0 of 4115 found in mempool. 4115 fetched through backend service.
Nov 10 11:06:58 [116] DEBUG: 0 websocket clients | 0 connected | 0 disconnected | (+0)
Nov 10 11:06:59 [116] DEBUG: Mempool blocks calculated in 0 seconds
Nov 10 11:06:59 [116] DEBUG: New block found (#816133)!
Nov 10 11:07:01 [116] DEBUG: Updated orphaned blocks cache. Found 13 orphaned blocks
Nov 10 11:07:03 [116] DEBUG: Indexing tx 1 of 4478 in block #816133
Nov 10 11:07:05 [116] DEBUG: Indexing tx 449 of 4478 in block #816133
Nov 10 11:07:06 [116] DEBUG: Indexing tx 897 of 4478 in block #816133
Nov 10 11:07:07 [116] DEBUG: Indexing tx 1345 of 4478 in block #816133
Nov 10 11:07:09 [116] DEBUG: Indexing tx 1793 of 4478 in block #816133
Nov 10 11:07:10 [116] DEBUG: Indexing tx 2241 of 4478 in block #816133
Nov 10 11:07:12 [116] DEBUG: Indexing tx 2689 of 4478 in block #816133
Nov 10 11:07:13 [116] DEBUG: Indexing tx 3137 of 4478 in block #816133
Nov 10 11:07:14 [116] DEBUG: Indexing tx 3585 of 4478 in block #816133
Nov 10 11:07:15 [116] DEBUG: Indexing tx 4033 of 4478 in block #816133
Nov 10 11:07:17 [116] DEBUG: Indexing tx 4478 of 4478 in block #816133
Nov 10 11:07:17 [116] DEBUG: 0 of 4478 found in mempool. 4478 fetched through backend service.
Nov 10 11:07:17 [116] DEBUG: 0 websocket clients | 0 connected | 0 disconnected | (+0)
Nov 10 11:07:17 [116] DEBUG: Mempool blocks calculated in 0 seconds
Nov 10 11:07:17 [116] DEBUG: New block found (#816134)!
Nov 10 11:07:19 [116] DEBUG: Updated orphaned blocks cache. Found 13 orphaned blocks
Nov 10 11:07:21 [116] DEBUG: Indexing tx 1 of 3483 in block #816134
Nov 10 11:07:21 [116] ERR: $updateBlocks stalled at "getting block data for 816134"
Nov 10 11:07:22 [116] DEBUG: Indexing tx 349 of 3483 in block #816134
Nov 10 11:07:23 [116] DEBUG: Memory usage: 0.10 GB / 6.39 GB
Nov 10 11:07:23 [116] DEBUG: Indexing tx 697 of 3483 in block #816134
Nov 10 11:07:24 [116] DEBUG: Indexing tx 1045 of 3483 in block #816134
Nov 10 11:07:25 [116] DEBUG: Indexing tx 1393 of 3483 in block #816134
Nov 10 11:07:27 [116] DEBUG: Indexing tx 1741 of 3483 in block #816134
Nov 10 11:07:28 [116] DEBUG: Indexing tx 2089 of 3483 in block #816134
Nov 10 11:07:29 [116] DEBUG: Indexing tx 2437 of 3483 in block #816134
Nov 10 11:07:30 [116] DEBUG: Indexing tx 2785 of 3483 in block #816134
Nov 10 11:07:32 [116] DEBUG: Indexing tx 3133 of 3483 in block #816134
Nov 10 11:07:32 [116] DEBUG: Indexing tx 3481 of 3483 in block #816134
Nov 10 11:07:32 [116] DEBUG: Indexing tx 3483 of 3483 in block #816134
Nov 10 11:07:32 [116] DEBUG: 0 of 3483 found in mempool. 3483 fetched through backend service.
Nov 10 11:07:33 [116] DEBUG: 0 websocket clients | 0 connected | 0 disconnected | (+0)
Nov 10 11:07:33 [116] DEBUG: Mempool blocks calculated in 0 seconds
Nov 10 11:07:33 [116] DEBUG: New block found (#816135)!
Nov 10 11:07:36 [116] DEBUG: Updated orphaned blocks cache. Found 13 orphaned blocks
Nov 10 11:07:38 [116] DEBUG: Indexing tx 1 of 4000 in block #816135
Nov 10 11:07:39 [116] DEBUG: Indexing tx 401 of 4000 in block #816135
Nov 10 11:07:41 [116] DEBUG: Indexing tx 801 of 4000 in block #816135
Nov 10 11:07:43 [116] DEBUG: Indexing tx 1201 of 4000 in block #816135
Nov 10 11:07:44 [116] DEBUG: Indexing tx 1601 of 4000 in block #816135
Nov 10 11:07:45 [116] DEBUG: Indexing tx 2001 of 4000 in block #816135
Nov 10 11:07:47 [116] DEBUG: Indexing tx 2401 of 4000 in block #816135
Nov 10 11:07:48 [116] DEBUG: Indexing tx 2801 of 4000 in block #816135
Nov 10 11:07:50 [116] DEBUG: Indexing tx 3201 of 4000 in block #816135
Nov 10 11:07:52 [116] DEBUG: Indexing tx 3601 of 4000 in block #816135
Nov 10 11:07:53 [116] DEBUG: Indexing tx 4000 of 4000 in block #816135
Nov 10 11:07:53 [116] DEBUG: 0 of 4000 found in mempool. 4000 fetched through backend service.
Nov 10 11:07:53 [116] DEBUG: 0 websocket clients | 0 connected | 0 disconnected | (+0)
Nov 10 11:07:53 [116] DEBUG: Mempool blocks calculated in 0 seconds
Nov 10 11:07:55 [116] DEBUG: New block found (#816136)!
Nov 10 11:07:58 [116] DEBUG: Updated orphaned blocks cache. Found 13 orphaned blocks
Nov 10 11:08:00 [116] DEBUG: Indexing tx 1 of 4644 in block #816136
Nov 10 11:08:01 [116] DEBUG: Indexing tx 465 of 4644 in block #816136
Nov 10 11:08:02 [116] DEBUG: Indexing tx 929 of 4644 in block #816136
Nov 10 11:08:04 [116] DEBUG: Indexing tx 1393 of 4644 in block #816136
Nov 10 11:08:05 [116] DEBUG: Indexing tx 1857 of 4644 in block #816136
Nov 10 11:08:07 [116] DEBUG: Indexing tx 2321 of 4644 in block #816136
Nov 10 11:08:08 [116] DEBUG: Indexing tx 2785 of 4644 in block #816136
Nov 10 11:08:09 [116] DEBUG: Indexing tx 3249 of 4644 in block #816136
Nov 10 11:08:11 [116] DEBUG: Indexing tx 3713 of 4644 in block #816136
Nov 10 11:08:13 [116] DEBUG: Indexing tx 4177 of 4644 in block #816136
Nov 10 11:08:15 [116] DEBUG: Indexing tx 4641 of 4644 in block #816136
Nov 10 11:08:15 [116] DEBUG: Indexing tx 4644 of 4644 in block #816136
Nov 10 11:08:15 [116] DEBUG: 0 of 4644 found in mempool. 4644 fetched through backend service.
Nov 10 11:08:15 [116] DEBUG: 0 websocket clients | 0 connected | 0 disconnected | (+0)
Nov 10 11:08:15 [116] DEBUG: Mempool blocks calculated in 0 seconds
Nov 10 11:08:16 [116] DEBUG: Updating mempool...
Nov 10 11:08:23 [116] DEBUG: Memory usage: 0.11 GB / 6.39 GB
Nov 10 11:09:23 [116] DEBUG: Memory usage: 0.62 GB / 6.39 GB
Nov 10 11:09:30 [116] DEBUG: fetched 9997 transactions
Nov 10 11:09:30 [116] DEBUG: Error finding 3 transactions in the mempool:
Nov 10 11:09:30 [116] DEBUG: Mempool is synchronizing. Processed 9997/139261 txs (7%)
Nov 10 11:09:48 [116] DEBUG: fetched 10000 transactions
Nov 10 11:09:48 [116] DEBUG: Mempool is synchronizing. Processed 19997/139261 txs (14%)
Nov 10 11:10:04 [116] DEBUG: fetched 10000 transactions
Nov 10 11:10:04 [116] DEBUG: Mempool is synchronizing. Processed 29997/139261 txs (22%)
Nov 10 11:10:16 [116] ERR: $updateMempool stalled at "fetched new transactions"
Nov 10 11:10:23 [116] DEBUG: Memory usage: 0.69 GB / 6.39 GB
Nov 10 11:10:32 [116] DEBUG: fetched 10000 transactions
Nov 10 11:10:32 [116] DEBUG: Mempool is synchronizing. Processed 39997/139261 txs (29%)
Nov 10 11:10:49 [116] DEBUG: fetched 10000 transactions
Nov 10 11:10:49 [116] DEBUG: Mempool is synchronizing. Processed 49997/139261 txs (36%)
Nov 10 11:11:05 [116] DEBUG: fetched 10000 transactions
Nov 10 11:11:05 [116] DEBUG: Mempool is synchronizing. Processed 59997/139261 txs (43%)
Nov 10 11:11:23 [116] DEBUG: Memory usage: 0.77 GB / 6.39 GB
Nov 10 11:11:33 [116] DEBUG: fetched 10000 transactions
Nov 10 11:11:33 [116] DEBUG: Mempool is synchronizing. Processed 69997/139261 txs (50%)
Nov 10 11:11:46 [116] DEBUG: fetched 10000 transactions
Nov 10 11:11:46 [116] DEBUG: Mempool is synchronizing. Processed 79997/139261 txs (57%)
Nov 10 11:12:01 [116] DEBUG: fetched 10000 transactions
Nov 10 11:12:01 [116] DEBUG: Mempool is synchronizing. Processed 89997/139261 txs (65%)
@krtk6160 does it crash on 2.5.0 too?
Yes it does, with the same OOM error code 137.
6.39GB should be more than enough for current mempool conditions, and from your logs it doesn't look like your mempool backend is using anywhere near that much memory anyway:
Nov 10 11:11:05 [116] DEBUG: Mempool is synchronizing. Processed 59997/139261 txs (43%) Nov 10 11:11:23 [116] DEBUG: Memory usage: 0.77 GB / 6.39 GB Nov 10 11:11:33 [116] DEBUG: fetched 10000 transactions Nov 10 11:11:33 [116] DEBUG: Mempool is synchronizing. Processed 69997/139261 txs (50%) Nov 10 11:11:46 [116] DEBUG: fetched 10000 transactions Nov 10 11:11:46 [116] DEBUG: Mempool is synchronizing. Processed 79997/139261 txs (57%) Nov 10 11:12:01 [116] DEBUG: fetched 10000 transactions Nov 10 11:12:01 [116] DEBUG: Mempool is synchronizing. Processed 89997/139261 txs (65%)
When nodejs exceeds the --max-old-space-size heap limit, it crashes itself with an error like:
<--- Last few GCs --->
[46017:0x130008000] 1625 ms: Mark-Compact (reduce) 254.7 (261.4) -> 254.0 (261.7) MB, 13.83 / 0.00 ms (+ 16.7 ms in 13 steps since start of marking, biggest step 6.6 ms, walltime since start of marking 33 ms) (average mu = 0.244, current mu = 0.264) [46017:0x130008000]
1668 ms: Mark-Compact (reduce) 255.0 (262.2) -> 254.5 (262.5) MB, 22.00 / 0.00 ms (+ 8.8 ms in 9 steps since start of marking, biggest step 3.7 ms, walltime since start of marking 33 ms) (average mu = 0.258, current mu = 0.272) al
<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
If you're seeing a Kubernetes Code 137 OOM error instead, that suggests to me that the container doesn't actually have 8GB of memory available for some reason, and is running out of space around 1-2GB of usage well before the nodejs heap limit kicks in.
Maybe double check your K8s pod/container config?
6.39GB should be more than enough for current mempool conditions, and from your logs it doesn't look like your mempool backend is using anywhere near that much memory anyway:
Nov 10 11:11:05 [116] DEBUG: Mempool is synchronizing. Processed 59997/139261 txs (43%) Nov 10 11:11:23 [116] DEBUG: Memory usage: 0.77 GB / 6.39 GB Nov 10 11:11:33 [116] DEBUG: fetched 10000 transactions Nov 10 11:11:33 [116] DEBUG: Mempool is synchronizing. Processed 69997/139261 txs (50%) Nov 10 11:11:46 [116] DEBUG: fetched 10000 transactions Nov 10 11:11:46 [116] DEBUG: Mempool is synchronizing. Processed 79997/139261 txs (57%) Nov 10 11:12:01 [116] DEBUG: fetched 10000 transactions Nov 10 11:12:01 [116] DEBUG: Mempool is synchronizing. Processed 89997/139261 txs (65%)
When nodejs exceeds the --max-old-space-size heap limit, it crashes itself with an error like:
<--- Last few GCs ---> [46017:0x130008000] 1625 ms: Mark-Compact (reduce) 254.7 (261.4) -> 254.0 (261.7) MB, 13.83 / 0.00 ms (+ 16.7 ms in 13 steps since start of marking, biggest step 6.6 ms, walltime since start of marking 33 ms) (average mu = 0.244, current mu = 0.264) [46017:0x130008000] 1668 ms: Mark-Compact (reduce) 255.0 (262.2) -> 254.5 (262.5) MB, 22.00 / 0.00 ms (+ 8.8 ms in 9 steps since start of marking, biggest step 3.7 ms, walltime since start of marking 33 ms) (average mu = 0.258, current mu = 0.272) al <--- JS stacktrace ---> FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memoryIf you're seeing a Kubernetes Code 137 OOM error instead, that suggests to me that the container doesn't actually have 8GB of memory available for some reason, and is running out of space around 1-2GB of usage well before the nodejs heap limit kicks in.
Maybe double check your K8s pod/container config?
I checked the pod limits and also changed them a few times and saw the change reflected in k8s and the gcloud console.
Blue is the actual usage, green the request and red the limit.