Out of memory when running tests

[zhemao]

I'm having quite a bit of trouble trying to get this to run. I am running on ArchLinux with Intel OpenCL. When I run make check, the tests fail with the message out of memory. I tried monitoring my memory usage while running run-tests, and in fact the memory does climb quite dramatically when the program is running. I think this might be a problem specific to Intel's OpenCL implementation, since the kernels are run on the CPU and hence memory for the kernels are allocated on the computer's normal RAM instead of GPU memory. I have 4 GB of RAM on my computer, and there is usually about 3 GB free under normal conditions. How large of an array are you allocating in your kernels?

[zhemao]

Actually, it doesn't seem to be related to the tests at all. Just running harlanc with no arguments causes a tremendous amount of memory usage, even though that doesn't even do anything.

[zhemao]

So, I've traced the memory issue down through the import statements, and I'm pretty sure it's coming from harlan/middle/language.scm. So something about all those define-language statements is causing petite to allocate a lot of memory.

[eholk]

Unfortunately, since I've started using @akeep's Nanopass Framework, the Harlan compiler uses a lot of memory, typically about 3GB. This sounds like you're probably running out of address space though. Are you using the 32-bit or 64-bit version of Petite? I suspect the fact that Harlan's memory usage is so high means it will require the 64-bit version.

[eholk]

I've just filed a related bug on the Nanopass framework: akeep/nanopass-framework#4

[zhemao]

I'm using the 64-bit version, so it's not address space. It's running out of memory because the physical memory is being used up and I don't have a swap partition. This only happens when I've got a few other applications running, though, so the compiler is still usable, although the user experience isn't that great due to the very long startup time. I feel like you could work around this problem by pre-expanding the define-language macros in some way.

[eholk]

Ah, I'm glad you're still able to run the compiler. I agree that the startup time isn't great, but hopefully we can improve it in the future. If you have full Chez Scheme, you can precompile the compiler, which means you just pay the startup cost once. Your idea of pre-expanding the define-language macros may be possible too, even with Petite.

On Sat, Aug 10, 2013 at 5:02 PM, Howard Mao [email protected]:

I'm using the 64-bit version, so it's not address space. It's running out of memory because the physical memory is being used up and I don't have a swap partition. This only happens when I've got a few other applications running, though, so the compiler is still usable, although the user experience isn't that great due to the very long startup time. I feel like you could work around this problem by pre-expanding the define-languagemacros in some way.

— Reply to this email directly or view it on GitHubhttps://github.com/eholk/harlan/issues/99#issuecomment-22449782 .

[akeep]

The problem with using the nanopass framework seems to be related to the number of record types that get created by each language. While that is not something that we can change, it might be possible to improve the memory usage (and with luck the performance) by changing how we create these records. Unfortunately, this is going to be a non-trivial change. (I've started thinking about how best to make this change, and hopefully, I'll get some time to hack on it today.)

In the meantime, there are two work arounds that might help. First, if you have access to the full Chez Scheme compiler, you can use that instead of petite. In my experiments, the high water mark was around 500MB instead of 3GB for petite. You can use it by changing line 6 in harlanc from:

petite --libdirs "$dir" --program "$schemeScript" $@

to

scheme --libdirs "$dir" --program "$schemeScript" $@

If you do not have the full Chez Scheme compiler, you can run harlanc using petite at optimize level 3, this causes less code to be generated for each of the records and reduced the memory usage for my petite process to peak at about 2.5 GB (which I recognize is still not great). You should also make sure you are using the 64-bit version of petite. You can do this as follows:

$ petite

Petite Chez Scheme Version 8.4.1
Copyright (c) 1985-2012 Cadence Research Systems

> (machine-type)
a6osx
> (fixnum-width)
61

The machine-type tells you information about the machine and operating system. Here the a6 indicates the amd64 or x86-64 machine and osx indicates I'm on an Mac running OS X. The fixnum-width is a little more direct, it will tell you the bit-width of integers minus the tag bits. In this case 61 indicates 64-bit, while 30 would indicate 32-bit.

You can change the harlanc script to use optimize level 3, by changing line 6 from:

petite --libdirs "$dir" --program "$schemeScript" $@

to

petite --libdirs "$dir" --optimize-level 3 --program "$schemeScript" $@

One thing to note is that optimize level 3 is petite's unsafe mode, so if something goes wrong inside the harlanc scheme code, you will not get a very nice error message, if you get an error message at all. (You may just get a core dump.) I'll try to get a fix into the nanopass framework as soon as possible, and hopefully it will help to fix things. (It is a bit of a shell game, since we still want the framework to perform the same work, we're just moving where that work happens, but hopefully we'll be able to find a way to do this that doesn't have the huge memory footprint).

[eholk]

@zhemao - If you enable a swap file, does that allow you to run the Harlan compiler?

[zhemao]

I can run the compiler successfully as long as I don't have other memory-intensive applications running. It does take a very long time though.

On Wed, Sep 4, 2013 at 12:47 PM, Eric Holk [email protected] wrote:

@zhemao https://github.com/zhemao - If you enable a swap file, does that allow you to run the Harlan compiler?

— Reply to this email directly or view it on GitHubhttps://github.com/eholk/harlan/issues/99#issuecomment-23805052 .

[eholk]

As of 6faa9f5b3895ddded7b4d1222e9a45ffe8fbb628, the memory usage and compile time should be significantly better, although there's still a lot more room for improvement.

[eholk]

I made another change, 417059e, which significantly reduces Harlan's memory usage. We should be able to further reduce it if necessary.