alpaca.cpp

alpaca.cpp copied to clipboard

Hello guys. First of all - Amazing work, both of you @antimatter15 and @ggerganov

I was playing with alpaca.cpp on my home computer (AMD 3400g, 16 Gb of RAM, Ubuntu 22.10) and figured it out in a hard way that even with the smallest 7B model it tends to eat out all the RAM after few shots and given I haven't configured both swap and out-of-memory-killer my linux just died out on me.

I was wondering, if there's any kind of memory limiter that could be set in place, an an optional parameter, so it will stay within the memory footprint just as a safety measure?

I don't mind to buy more RAM or configure swap - not at all, but if there's some kind of memory leak is happening, it's better to get that to know in some easy way, rather than the way I just did :)

Yeah, I can confirm this.

Yesterday I was testing it on commit 99f3908c515c73c7acbda307565234f33ec9738d from master. Today I have updated to the latest commit 7636ddb9e45807ce4317717f75a4d7ce5d23541a and rebuilt it.

At the startup, the memory footprint for 7B model is ~ 4.1 gigs, but with the number of interactions it grows by ~ 100M with every prompt. Is it a memory leak we have here?

I can confirm that I see the similar pattern of memory leaking, and every token takes about 20M for 13B model.

This happens at alpaca.cpp, while the latest version of llama doesn't.