monorepo
monorepo copied to clipboard
opral
git proxy server running out of memory with badges
Problem
Opening large repositories in the editor like https://inlang.com/editor/github.com/NiklasBuchfink/jitsi-meet can lead to out-of-memory errors.
Jun 20 11:46:09 AM FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
Reproduction
- Unsure. Opening jisti-meet works for me but @NiklasBuchfink seemed to have triggered this bug by opening the repo.
- But the logs in render.com can be checked https://dashboard.render.com/ (ignore "wrong inlang config setup failed")
Additional information
This issue is likely part of removing the isomorphic git express middleware and having a dedicated lisa/server.
@samuelstroschein Request the following badge triggers this issue: https://inlang.com/badge?url=github.com/NiklasBuchfink/jitsi-meet
@NiklasBuchfink Request the following badge triggers this issue: https://inlang.com/badge?url=github.com/NiklasBuchfink/jitsi-meet
Oh, I see. In that case, here's whats likely happening:
- N Parallel requests for the large repository (300MB) badge hit the server
- no badge is cached yet
- the server tries to clone N times the large repository in parallel into memory to generate the badge
- N > 5 comes down to 1.8GB of RAM + other tasks = exceeds memory.
This bug should be auto-fixed by lazy cloning of repos. The entire repo wouldn't get cloned into memory anymore. Massively speeding up the generation of badges, increasing cache hit chance, and reducing memory consumption.
@araknast really looking forward to lazy cloning/only download file contents on demand
The future plan of project lisa foresees an unbundling of inlang.com and the git proxy server. A crash in a lisa server won't take down inlang.com anymore.
@NiklasBuchfink @samuelstroschein Whats the status on this issue? Will @inlang/lix handle this? Who should be assinged?
@felixhaeberle this will be one of my immediate next tasks and i allready started looking into it, first step split up the systems, second step fix possible memory issues with proxy server
@felixhaeberle the full fix for this is the lazy loading lix feature, martin shared a loom today with the current progress on this front. the planned timeline is attached, please note, that it will be probably be a week later or so as the editor changes and edge cases to work with the github app/ new oauth scope logic were a bit more fiddly than in the project roadmap: