Issues with Monte Carlo Tree Search on alpaca.cpp

[simsim314]

So I've managed to run alpaca.cpp from Visual Studio, and it all looks great and interesting.

But then I see in code that you provide the: llama_eval eval function here: https://github.com/antimatter15/alpaca.cpp/blob/master/chat.cpp#L954

You pass embd of length one during the generation of content. I placed prints there it was printing 1.

So I guess all the data is stored inside the model class. With model.ctx or wtr, I am not sure how it all works yet.

Anyway for Monte Carlo Tree Search I sometimes want to choose most prospective state of the model class. So I need Save/Load functions of this state. I can start everything from scratch again passing all data token by token, but this is heavy function it takes most of the time of the run, so I don't want to waste time to increase the length only to the best candidates, but it takes a lot of memory, so I need to save the state to a disk. Please advice...

I could run simpler MCTS to begin with, maybe it's what I will do first, or waste time only on the best candidates only. But I would like to be able to Save/Load the state to file, from the middle of the run, and continue from there. The basic idea is to try reproduce the RL with chess/GO engines MCTS. They say the function there playing very weak, but with MCTS it's like magic. It could be the case for llama models, many people would just wait a day or two for a slower response, but for free and of quality of GPT4.