Reduce memory consumption of Silver compiler

[tedinski]

We probably need some memory usage profiling done again.

Presently, full builds w/ MWDA are hitting 2G memory usage. Normally, I wouldn't care about memory usage too much (it's hardly a priority for us) but this impacts our ability to build things in parallel with Jenkins. The machine has 16G, which means we realistically have about 13G to play with. Compiles hitting 2G means we've had to drop down to 6 executors to keep things from swapping.

The machine has 12 (real) cores, so I think we should target 1G compiles, so we could have a full 12 executors. I'm not sure how much work this would be to accomplish.

Goals:

• [ ] Reduce memory usage (MWDA on Silver, AbleC+exts) to 1G, at most 1.5G.
• [x] Increase executor count to 12, at least 8
• [ ] Reduce silver script to -Xmx 1500M, or at least back to -Xmx 2000M

Candidates for improvement:

• The obvious worst offender is the Silver compilation process: we parse everything, check everything, translate everything. Breaking things up into strongly connected components and building each in turn would be nice. This would be an unfortunate amount of work at the moment though.
• Introduce arrays in addition to lists. Lists are often "fine" in many cases, but certainly everything we traverse multiple times would benefit from being an array instead.
• MWDA code: we do a lot of iteration until fixed point, and this pins a lot of data in memory. We could try rewriting some of this imperatively, or we could try some sort of hacky 'deepseq' approach to pruning thunks.
• Type checking code: there's a lot of inefficiency here, it was never written to be efficient
• Env code: Involves an unfortunate amount of allocation and copying, I believe.
• Stdlib: we could try providing native java implementations of certain key functions.
• Runtime & compiler improvements: we've done almost no work on eliminating thunk allocation, that could help.

[tedinski]

Ug, executor count dropped to 5. :(

[krame505]

What I don't get is why builds are actually sometimes failing from this, as opposed to simply swapping and running a bit slower?

[tedinski]

Well, they'd run more than a bit slower.

I believe the JVM deliberately attempts to ensure that amount of JVM-allocated memory is actually available on the host. It's been awhile since I remember actually looking into its source code to see how it allocates, but as I recall, it goes out of its way to try to get Linux to admit failure if it's short on RAM, instead of just blindly charging forward.

[krame505]

... and it happened again: http://coldpress.cs.umn.edu:8080/job/melt-umn/job/ableC-sqlite/job/feature%252Fenv/15/consoleFull

For some reason it only seems to be the sqlite extension acting up though. Not sure what is special about this one in particular.

[tedinski]

Ah, that's not the machine running out of memory! That's that JVM process running out. We need to just up its -Xmx line for something there. Let me see...

[tedinski]

Yeah, so, if you want to bump up the silver script default to 2500M or so, that'd probably take care of it for now.

We don't want to increase this number too much, it puts a minimum system requirements on people that's embarrassing. 2.5G means a vm should have at least 4G, which means the host should now have more than that, which practically speaking means 8G these days, and yikes.

[krame505]

Just changed it, Jenkins is building right now. We're probably fine going back to 6 executors, since after watching it for a bit we seem to be staying significantly under maximum CPU and memory utilization.

[krame505]

Semi-fixed on the Jenkins front with the move to foundry with oodles of RAM, but we would still like to reduce Silver's memory usage in general, and the above items that Ted mentioned are all reasonable candidates.