Not having enough memory just causes a segfault or something

[LoganDark]

So. I'm trying to build with CMake on Windows 11 and the thing just stops after it's done loading the model.

And apparently, this is a segfault.

Yay yay yyayy yyayay

this is a memory allocation failure it seems, from me not having enough memory. not like llama.cpp Tells Me That lmao, it just segfaults

(ctx->mem_buffer is nullptr which probably means the malloc just failed)

[LoganDark]

can confirm clearing tons of memory seems to make it work:

[LoganDark]

Performing the quantization step from f16 down to q4_0 significantly helps with the memory usage (I am dumb but eventually figured this out even though I was supposed to do it anyway)

It does make the model extra drunk though (which I guess is to be expected from the 7B version)

[gadzbi123]

I was wondering why the q4_0 was giving such a wierd responses

[LoganDark]

I was wondering why the q4_0 was giving such a wierd responses

q4_0 will say 2 + 2 = 10, then proceeds to explain how that's because they are in their "respective ranges" of "-5 through 8" by not being "greater than 9 or lesser than -4", but that's also an off-by-one error, not like it matters because what?

It's actually hilarious

[tjohnman]

Detecting when a stack overflow is going to happen in a portable way is tricky. If we could use malloc() to allocate this memory it would be able to fail gracefully.

[LoganDark]

Detecting when a stack overflow is going to happen in a portable way is tricky.

Is this a stack overflow? ctx->mem_buffer is null. Clearly an actual allocation failed first and the segfault happened because of an attempt to access something through the resulting null pointer. Which could have been detected earlier and handled cleanly without segfaulting (i.e. printing an error message and exiting)

[tjohnman]

You're right, my bad. In that case it should be easier to deal with.

[sw]

There is now an assert that checks mem_buffer, even in non-debug builds: https://github.com/ggerganov/llama.cpp/blob/173d0e6419e8f8f3c1f4f13201b777f4c60629f3/ggml.c#L4571

Closing this as it's quite old, please re-open if you still encounter the problem with a recent revision.

[LoganDark]

There is now an assert that checks mem_buffer, even in non-debug builds:

Yep, that looks like it would fix it~

Closing this as it's quite old

Actually you'd be closing it because it's solved lol