Large projects causing memory error?

[ryantuck]

Hey @fabio-looker - I think we uncovered a bug whereby lookml-parser fails on large (~800 files) projects: https://github.com/WarbyParker/lookmlint/issues/50#issuecomment-447420100

Didn't know if you had seen that before, or had any immediate ideas on how one would go about fixing that. I'll poke around in the code myself to see if anything obvious jumps out, but javascript isn't quite my strength. I've got a handful of ideas for working around it for lookmlint in the meantime, but wanted to flag this as an issue 👍

[fabio-looker]

Thanks for the report! I've run it on pretty large projects without issue, but I'll have to dig it up and try it again.

That said, it is true that all the files are loaded into memory at once and at some point it will break. Shouldn't be that hard to refactor it to default to in-memory but have some sort of file-system fallback mode if there are too many objects in the project

[kingfink]

Thanks @fabio-looker and @ryantuck ! If you need an example happy to provide...or if I have some free time in the near future I'll try poking around

[fabio-looker]

Wouldn't hurt to have a realistic project for repro'ing. Can you send it to [email protected] ?

[kingfink]

Good news! I was able to get this working 🙂

I needed to tweak my node options: export NODE_OPTIONS=--max_old_space_size=4096

[fabio-looker]

Glad to hear it. I'm going to keep the ticket open as I think this could be addressed anyway

[fabio-looker]

While this issue is not fundamentally resolved, 5.0.0 should reduce memory usage due to less repetitive representations being built out in-memory