Build error: `parseAndElaborate starting: Out of memory. Unable to allocate 804,197,972 bytes.`

[barracuda156]

What could be done about this? This is in VM, but it has 10 GB RAM and I had mlton 20230721 building fine there. But 20240119 does not.

cp -fpR front-end/mlb.grm.sml front-end/mlb.grm.sml.boot
/opt/local/bin/gsed -e 's/in Array.fromList(List.map actionRowLookUp actionRowNumbers)/in Array.fromList(Pervasive.List.map actionRowLookUp actionRowNumbers)/' front-end/ml.grm.sml.in > front-end/ml.grm.sml
chmod -w front-end/mlb.grm.*
rm -rf front-end/ml.grm.sml.in
cp -fpR front-end/ml.grm.sml front-end/ml.grm.sml.boot
chmod -w front-end/ml.grm.*
rm -rf front-end/ml.lex.sml.in
cp -fpR front-end/ml.lex.sml front-end/ml.lex.sml.boot
chmod -w front-end/ml.lex.*
Compiling mlton
"/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_mlton/mlton/work/mlton-2637b22ccdd9cd1268b6b46ff6107066bf8c8888/build/bin/mlton" \
		@MLton ram-slop 0.7  gc-summary -- \
		 -default-ann 'sequenceNonUnit warn' -default-ann 'warnUnused true' -const 'MLton.debug false' -verbose 2 	\
		-target self -output mlton-compile			\
		mlton.mlb
MLton 20240119 starting
   Compile SML starting
      frontend starting
         parseAndElaborate starting
Out of memory.  Unable to allocate 804,197,972 bytes.

make[2]: *** [mlton-compile] Error 1
make[2]: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_mlton/mlton/work/mlton-2637b22ccdd9cd1268b6b46ff6107066bf8c8888/mlton'
make[1]: *** [compiler] Error 2

[MatthewFluet]

Can you provide more details? The architecture and os where the failure is being observed?

It looks like this is a self-compile that is failing. The "Unable to allocate 804,197,972 bytes." is far from 10GB, and still reasonably far from the 4GB address space limit of a 32-bit platform. Since 20230721, there has been one change that might increase the memory usage on a 32-bit platform: 519aacb69421c98c1329edb9a3a62ad63e667e3b, which forces the alignment of objptrs within sequene objects, fixing a bug with the mark-compact GC.

[barracuda156]

@MatthewFluet Thank you for responding!

This is on 10.6.8, building for ppc (via Rosetta).

P. S. I have built mlton 20240119 natively on 10.6 ppc, so it should not be a problem of the arch as such. But for w/e reason Rosetta build fails. I will try to revert that commit and see if it helps.

[MatthewFluet]

You might try compiling using make MLTON_RUNTIME_ARGS="gc-messages", which will dump a lot of details about GC activity. The 804,197,972 bytes is a fair bit smaller than the 4GB address space of a 32-bit platform, but it does need to be contiguous. I wonder if going through Rosetta leads to more fragmentation in the virtual address space. Perhaps it was nearing the limit and the extra alignment pushes it over the edge.

Yet another thing to try would be to compile with make MLTON_RUNTIME_ARGS="may-page-heap true ", which allows the runtime system to write the heap to disk (and then deallocate the heap memory) in the hopes of making a large contiguous region of memory available. It's disabled by default, because it is very slow and it is usually more helpful to simply die with an out of memory panic than repeatedly paging the heap to disk as the heap size slowly increases.

[MatthewFluet]

The other interesting thing about your log is that that error message comes from:

https://github.com/MLton/mlton/blob/2637b22ccdd9cd1268b6b46ff6107066bf8c8888/runtime/gc/virtual-memory.c#L10-L20

and, on darwin, GC_displayMem is implemented as

https://github.com/MLton/mlton/blob/2637b22ccdd9cd1268b6b46ff6107066bf8c8888/runtime/platform/darwin.c#L12-L17

Yet, we're not seeing the output of that command.

[barracuda156]

@MatthewFluet Weirdly, this time it built successfully. The only change I made was adding build.args-append MLTON_RUNTIME_ARGS="gc-messages" into portfile, but this is not supposed to change the build itself, as I understood, only deliver extra debug output. Maybe DM got short of disk space last time or some random factor?

macmini:~ svacchanda$ port -v installed mlton | grep active
  mlton @20240119_0 (active) requested_variants='' platform='darwin 10' archs='ppc' date='2024-02-16T23:03:44+0800'

[barracuda156]

Here is the build log: mlton_10.6.8_rosetta.txt

[MatthewFluet]

Closing, since not reproducible.