Out of memory error when using ETag

[azigit]

I'd love to use the server on my low-ram (512MB) vps. It works nicely with smaller files, but when attempting to download a 200MB file it fails: 'fatal error: runtime: out of memory'. Is there any solution to this? Thanks.

[patrickdappollonio]

Hey!

Sounds like an interesting situation. Would you mind sharing the entire stack trace if any of the crash?

I'll try to reproduce on my end.

[azigit]

Can you explain how to obtain the stack trace?

[patrickdappollonio]

When the app crashed, it should've printed a few more lines to the terminal after the out of memory. If not, what might've happened is that your OS might have decided to kill the app preemptively.

If it's a crash in our end, it's something we can trace to a line of code, then optimize it so it does everything in small chunks rather than dumping it all to memory.

[patrickdappollonio]

I took a look at this and I have some bad news: you probably didn't even get a stack trace because it wasn't Go what finished your application, it was most likely your OS with an oomkilled. That won't leave a stack trace.

I assume though the issue comes from this line: https://github.com/patrickdappollonio/http-server/blob/v2.1.2/internal/server/handlers.go#L164

Thing with that is... I didn't write it :rofl: it was written by the Go authors and making a copy of it while keeping all features is a daunting task: they implemented a lot of the RFCs for the HTTP protocol, including range requests, seeking and all.

I could clone it and then improve upon what they wrote but I'm not sure how better I am than the Go authors at improving the reliability of the code.

I will look around for some other options. The newer versions of Go allow you to control, to an extent, RAM usage, so it might be something we can document or "hint" at the Go code so it runs within the limits of your VPS.

Will keep you posted!

[azigit]

Patrick: No - I did get a trace, I just wasn't sure whether there was anything else that you'd need. So I ran it again copied the output to a file which is attached. serv-out-2.txt Thanks for looking into this.

[patrickdappollonio]

Interesting! It seems to hint at the etag generation taking a chunk of your ram.

This might be nothing but you can try with --disable-etag and see if it makes any difference.

Some of the other buffers though are related to what I mentioned above, which I can't really control 😔

[azigit]

Works! Thanks very much for your help.

[patrickdappollonio]

Awesome! I'll keep this issue open, update it a bit to reflect the etag memory consumption and I think that's something we can definitely improve here 😁

Thanks a lot for reporting it!