ord icon indicating copy to clipboard operation
ord copied to clipboard

Investigate Savepoints not being released by OS process and causing memory leak

Open raphjaph opened this issue 9 months ago • 3 comments

raphjaph avatar Sep 09 '23 17:09 raphjaph

Hey - is this still a potential issue ? I'm experiencing ord server dying frequently due to memory problems.

Environment;

  • ubuntu latest
  • ord 0.18.3
  • 16G ram server
  • swap 2560 MB - have increased memory to investigate

I captured this from syslog;

2024-04-26T15:13:52.500397+00:00 spinoza kernel: [99407.410642] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-211.scope,task=ord,pid=167904,uid=1000
2024-04-26T15:13:52.500416+00:00 spinoza kernel: [99407.410772] Out of memory: Killed process 167904 (ord) total-vm:20672532kB, anon-rss:15660632kB, file-rss:384kB, shmem-rss:0kB, UID:1000 pgtables:38624kB oom_score_adj:0
2024-04-26T15:13:52.570945+00:00 spinoza systemd[1]: session-211.scope: A process of this unit has been killed by the OOM killer.
2024-04-26T15:13:54.312137+00:00 spinoza systemd[1]: session-211.scope: Deactivated successfully.

The process id is the ord running as ( note i set height-limit random high to avoid the default of 1000 - don't believe this is a factor but not sure).

nohup ord --data-dir $HOME/ordinals  --index-runes --height-limit 1000000 server --http-port 8080 &

Other settings;

  "commit_interval": 5000,
  "first_inscription_height": 767430,
  "height_limit": null,
  "hidden": [],
  "index_cache_size": 4190053376,

radicleart avatar Apr 26 '24 15:04 radicleart

Yes, this probably still is a problem. It's hard to reproduce and only occurs on some systems and configurations. Is your Bitcoin Core node fully synced?

raphjaph avatar May 07 '24 13:05 raphjaph

Yep - bitcoin synced on mainnet.

I've managed to work around by grabbing a snapshot of the index here the server synced from there.

radicleart avatar May 07 '24 15:05 radicleart