MLTQ

Same here- doing a lambda function on a large dataframe hangs at seemingly random points, some threads run to completion and others fail. try-catching in the lambda function doesn't seem...

Hi, I wanted to do this too and figured out a way- pretty rudimentary but it works! Add this function to cli.py then run it and it will pop up...

M1 Pro 32GB, 30B model: main: mem per token = 43387780 bytes main: load time = 10701.85 ms main: sample time = 279.92 ms main: predict time = 37065.80 ms...

Sounds like the ideal structure of this would be to load the model into memory in interactive mode, listen for input on some port, then wait for initial prompt &...

Thanks @nschulzke, I have been doing this, but I would ideally want to do something like: `./main` and it look for a config file, and if it finds it, use...

You can run the model in chat mode and suppress diagnostic output, seems like it wouldn't be too difficult to integrate.