Improve out-of-memory error during fasm generation (seen in nexys_video counter_test example)

[tcal-x]

trafficstars

My laptop has 10GB available according to /proc/meminfo. I see this error running the counter_test example with TARGET=nexys_video:

Serial number (magic cookie) for the routing is: 416094740
Circuit successfully routed with a channel width factor of 500.
Incr Slack updates 1 in 1.4106e-05 sec
Full Max Req/Worst Slack updates 1 in 4.114e-06 sec
Incr Max Req/Worst Slack updates 0 in 0 sec
Incr Criticality updates 0 in 0 sec
Full Criticality updates 1 in 1.8798e-05 sec
Writing Implementation FASM: top.fasm
/media/tim/GIT/opt/symbiflow/xc7/install/bin/vpr_common: line 116: 2752047 Killed                  genfasm ${ARCH_DEF} ${EBLIF} --device ${DEVICE_NAME} ${VPR_OPTIONS} --read_rr_graph ${RR_GRAPH} $@
make: *** [Makefile:54: build/nexys_video/top.fasm] Error 137

The file build/nexyx_video/top.fasm is left behind. Is it partially written? If I run the make command again, top.bit is generated from top.fasm. Is the top.bit file correct?

I have two concerns here:

• The user should get a clear "out of memory" error if that is indeed the case.
• When an error is encountered while generating top.fasm, that file should be removed, or else a subsequent make will use it and possibly write an incorrect top.bit file.

Note: If I close other applications to get 15GM of available memory, the flow completes without error, although I haven't had a chance to test the generated bitstream on the Nexys Video board yet.

[mithro]

@litghost - Ideas?

[mithro]

Makefiles do have a remove file on failure option?

[litghost]

Makefiles do have a remove file on failure option?

We need to add .DELETE_ON_ERROR to the makefile. @tcal-x, can you please retest your case with .DELETE_ON_ERROR in the makefile?

[litghost]

Link to relevant documentation: https://www.gnu.org/software/make/manual/html_node/Errors.html#Errors

Errors (GNU make)

Errors (GNU make)

[tcal-x]

@litghost , yep, that works. I'll open a PR with the change.

I could also add a message on failure, "If you see 'Killed...' above, the process may have been unable to allocate sufficient memory".

[andrewb1999]

Just as an FYI, I run into this same issue in yosys when running very large designs and I have another memory intensive application open in the background. It would be helpful to add some warning there as well.

[tcal-x]

@kgugala @mithro I'd like to add this to the cleanup list. It still dies with just "Error 137", no mention of out-of-memory. This is running the counter example; I have 11GB free but it's not enough:

Serial number (magic cookie) for the routing is: 481366680
Circuit successfully routed with a channel width factor of 500.
Incr Slack updates 1 in 1.5997e-05 sec
Full Max Req/Worst Slack updates 1 in 5.446e-06 sec
Incr Max Req/Worst Slack updates 0 in 0 sec
Incr Criticality updates 0 in 0 sec
Full Criticality updates 1 in 1.992e-05 sec
Writing Implementation FASM: top.fasm
/media/tim/GIT/opt/sf2/xc7/install/bin/vpr_common: line 116: 1737994 Killed                  genfasm ${ARCH_DEF} ${EBLIF} --device ${DEVICE_NAME} ${VPR_OPTIONS} --read_rr_graph ${RR_GRAPH} $@
make: *** [Makefile:57: build/nexys_video/top.fasm] Error 137
make: *** Deleting file 'build/nexys_video/top.fasm'