haskell-language-server icon indicating copy to clipboard operation
haskell-language-server copied to clipboard
verified publisher icon haskell

Possible memory leak (HLS got OOM-killed after being idle for a while)

Open gfarrell opened this issue 1 year ago • 6 comments

Your environment

  • Which OS do you use? Archlinux on 6.6.60-1-lts x86_64
  • Which version of GHC do you use and how did you install it? 9.8.2 via nix flake
  • How is your project built? cabal build (multi-component due to test suites etc.)
  • Which LSP client (editor/plugin) do you use? neovim + built-in LSP client + nvim-lspconfig
  • Which version of HLS do you use and how did you install it? 2.9.0.0 via nix flake
  • Have you configured HLS in any way (especially: a hie.yaml file)? No, only set the formatting programmes via lspconfig

Steps to reproduce

I had been working in a project, left my laptop suspended for a few days, reopened it, was doing something else (in a web browser), hadn't opened the project, then the system froze and the OOM killer killed HLS.

Expected behaviour

I would not expect HLS memory requirements to keep growing -- this is not an enormous project (see cloc stats below). There are five components (library, executable, three tests suites, and a debug executable).

-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Haskell                        159           1834           2394          10023

Actual behaviour

HLS stats just seem totally off (orders of magnitude greater than every other task).

Out of memory: Killed process 255537 (.haskell-langua) total-vm:1080298952kB, anon-rss:25930364kB, file-rss:544kB, shmem-rss:0kB, UID:1000 pgtables:155304kB oom_score_adj:0

Debug information

The only logs I have from today for HLS are the following:

[ERROR][2024-11-27 10:12:01] .../vim/lsp/rpc.lua:770	"rpc"	"/nix/store/9aid4qjxi2ms1f4ir8688r2wk72zcqs2-haskell-language-server-2.9.0.0/bin/haskell-language-server-wrapper"	"stderr"	"2024-11-27T10:12:01.968611Z | Info | Live bytes: 3632.80MB Heap size: 6275.73MB\n"
[ERROR][2024-11-27 10:13:02] .../vim/lsp/rpc.lua:770	"rpc"	"/nix/store/9aid4qjxi2ms1f4ir8688r2wk72zcqs2-haskell-language-server-2.9.0.0/bin/haskell-language-server-wrapper"	"stderr"	"2024-11-27T10:13:02.026863Z | Info | Live bytes: 6875.76MB Heap size: 8943.30MB\n"
[ERROR][2024-11-27 10:14:02] .../vim/lsp/rpc.lua:770	"rpc"	"/nix/store/9aid4qjxi2ms1f4ir8688r2wk72zcqs2-haskell-language-server-2.9.0.0/bin/haskell-language-server-wrapper"	"stderr"	"2024-11-27T10:14:02.043215Z | Info | Live bytes: 12095.45MB Heap size: 14383.32MB\n"
[ERROR][2024-11-27 10:15:02] .../vim/lsp/rpc.lua:770	"rpc"	"/nix/store/9aid4qjxi2ms1f4ir8688r2wk72zcqs2-haskell-language-server-2.9.0.0/bin/haskell-language-server-wrapper"	"stderr"	"2024-11-27T10:15:02.100070Z | Info | Live bytes: 18051.79MB Heap size: 20578.30MB\n"
[ERROR][2024-11-27 10:16:02] .../vim/lsp/rpc.lua:770	"rpc"	"/nix/store/9aid4qjxi2ms1f4ir8688r2wk72zcqs2-haskell-language-server-2.9.0.0/bin/haskell-language-server-wrapper"	"stderr"	"2024-11-27T10:16:02.160041Z | Info | Live bytes: 23018.72MB Heap size: 25846.35MB\n"
[ERROR][2024-11-27 10:17:02] .../vim/lsp/rpc.lua:770	"rpc"	"/nix/store/9aid4qjxi2ms1f4ir8688r2wk72zcqs2-haskell-language-server-2.9.0.0/bin/haskell-language-server-wrapper"	"stderr"	"2024-11-27T10:17:02.220165Z | Info | Live bytes: 29151.49MB Heap size: 32224.84MB\n"
[ERROR][2024-11-27 10:18:02] .../vim/lsp/rpc.lua:770	"rpc"	"/nix/store/9aid4qjxi2ms1f4ir8688r2wk72zcqs2-haskell-language-server-2.9.0.0/bin/haskell-language-server-wrapper"	"stderr"	"2024-11-27T10:18:02.280180Z | Info | Live bytes: 35901.97MB Heap size: 39245.05MB\n"
[ERROR][2024-11-27 10:19:02] .../vim/lsp/rpc.lua:770	"rpc"	"/nix/store/9aid4qjxi2ms1f4ir8688r2wk72zcqs2-haskell-language-server-2.9.0.0/bin/haskell-language-server-wrapper"	"stderr"	"2024-11-27T10:19:02.340127Z | Info | Live bytes: 41172.92MB Heap size: 44731.20MB\n"
[ERROR][2024-11-27 10:20:02] .../vim/lsp/rpc.lua:770	"rpc"	"/nix/store/9aid4qjxi2ms1f4ir8688r2wk72zcqs2-haskell-language-server-2.9.0.0/bin/haskell-language-server-wrapper"	"stderr"	"2024-11-27T10:20:02.400208Z | Info | Live bytes: 47085.01MB Heap size: 50880.05MB\n"
[ERROR][2024-11-27 10:21:02] .../vim/lsp/rpc.lua:770	"rpc"	"/nix/store/9aid4qjxi2ms1f4ir8688r2wk72zcqs2-haskell-language-server-2.9.0.0/bin/haskell-language-server-wrapper"	"stderr"	"2024-11-27T10:21:02.460071Z | Info | Live bytes: 52952.92MB Heap size: 56982.77MB\n"
[ERROR][2024-11-27 10:22:02] .../vim/lsp/rpc.lua:770	"rpc"	"/nix/store/9aid4qjxi2ms1f4ir8688r2wk72zcqs2-haskell-language-server-2.9.0.0/bin/haskell-language-server-wrapper"	"stderr"	"2024-11-27T10:22:02.520156Z | Info | Live bytes: 58423.89MB Heap size: 62673.39MB\n"
[ERROR][2024-11-27 10:23:02] .../vim/lsp/rpc.lua:770	"rpc"	"/nix/store/9aid4qjxi2ms1f4ir8688r2wk72zcqs2-haskell-language-server-2.9.0.0/bin/haskell-language-server-wrapper"	"stderr"	"2024-11-27T10:23:02.580138Z | Info | Live bytes: 63497.64MB Heap size: 67949.82MB\n"
[ERROR][2024-11-27 10:24:02] .../vim/lsp/rpc.lua:770	"rpc"	"/nix/store/9aid4qjxi2ms1f4ir8688r2wk72zcqs2-haskell-language-server-2.9.0.0/bin/haskell-language-server-wrapper"	"stderr"	"2024-11-27T10:24:02.614662Z | Info | Live bytes: 68659.72MB Heap size: 73318.53MB\n"
[ERROR][2024-11-27 10:25:02] .../vim/lsp/rpc.lua:770	"rpc"	"/nix/store/9aid4qjxi2ms1f4ir8688r2wk72zcqs2-haskell-language-server-2.9.0.0/bin/haskell-language-server-wrapper"	"stderr"	"2024-11-27T10:25:02.694281Z | Info | Live bytes: 74483.63MB Heap size: 79376.15MB\n"

gfarrell avatar Nov 27 '24 10:11 gfarrell

Hi, thank you for the bug report! Can you reproduce the issue? Can you link the project the issue is reproducible on? Otherwise, can you provide the full logs of the run? If possible, pass --debug to HLS to get even more logs.

At the moment, we are not aware of a memory leak in HLS and the size of your project is indeed tiny.

fendor avatar Nov 27 '24 12:11 fendor

Hi @fendor:

  1. I have not yet been able to reproduce the issue but I will pay attention to see if there are any other similar incidents. As mentioned in the issue description, I hadn't worked on it for a few days, but then memory usage spiralled and the OOM killer stepped in.
  2. The project is proprietary, so I can't link it.
  3. Here is the full log for the run (after many restarts of HLS in that session as it kept getting upset)

hls.log

gfarrell avatar Nov 27 '24 16:11 gfarrell

Hi @fendor:

  1. I have not yet been able to reproduce the issue but I will pay attention to see if there are any other similar incidents. As mentioned in the issue description, I hadn't worked on it for a few days, but then memory usage spiralled and the OOM killer stepped in.
  2. The project is proprietary, so I can't link it.
  3. Here is the full log for the run (after many restarts of HLS in that session as it kept getting upset)

hls.log

May I know how to get log when I´m using Neovim+Haskell.vim+ Haskell LSP ? I had same memory leak issue using LSP: HLS 2.9.0.1 GHC 9.8.2

yellowbean avatar Dec 10 '24 09:12 yellowbean

I just do :LspLog

gfarrell avatar Dec 10 '24 11:12 gfarrell

@gfarrell Thanks for the logs, but I am afraid they don't contain much information. Can you run your HLS executable with the --debug flag and link the logs?

fendor avatar Dec 10 '24 14:12 fendor

@fendor I am a haskell beginner, also on arch linux. I had a similar sounding thing occur. Had claude help me debug it because i didn't have an OOM killer installed so i did get a freeze and had to reboot.

Additional Data Point: HLS 2.12.0.0 + GHC 9.8.4 OOM

I experienced what appears to be the same issue.

Environment

  • OS: Arch Linux (kernel 6.17.9-arch1-1)
  • Hardware: Framework Laptop 16 (AMD Ryzen 7040 Series), 64GB RAM
  • HLS: 2.12.0.0
  • GHC: 9.8.4
  • Editor: Neovim with haskell-tools.nvim
  • HLS config: checkProject = true

What happened

I was watching a video stream (not actively coding). HLS was running in the background with a small Servant project open (~10 source files). The system froze completely and required a hard reboot.

OOM Killer logs

Timer invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=100

Out of memory: Killed process 654102 (ghc-9.8.4) total-vm:1074341916kB, anon-rss:58544340kB, file-rss:320kB, shmem-rss:0kB, UID:1000 pgtables:120632kB oom_score_adj:0

oom_reaper: reaped process 654102 (ghc-9.8.4), now anon-rss:0kB, file-rss:320kB, shmem-rss:0kB

Key numbers

Metric Value
Virtual memory 1,074 GB (1 TB)
RSS (actual RAM) 58.5 GB
Page tables 120 MB
Swap at OOM 120 KB free of 4 GB

Memory state at crash

Mem-Info: active_anon:7775105 inactive_anon:7687823 isolated_anon:0 Free swap = 120kB Total swap = 4177916kB

Notes

  • I was not actively building or editing - just had a Haskell file open
  • The project uses Servant/Aeson/Warp (standard web stack)
  • GHC 9.8.4 deps were not cached (different from my other projects using 9.10.3)
  • The 1TB virtual memory allocation suggests runaway/unbounded allocation rather than normal compilation I did have checkProject = true on also, i have since disabled it.

ryanbas21 avatar Nov 30 '25 04:11 ryanbas21