Georgi Gerganov

llama.cpp + Final Jeopardy

5

I was browsing reddit and saw this post: https://www.reddit.com/r/LocalLLaMA/comments/12xkm9v/alpaca_vs_final_jeopardy/ If anyone is interested, it would be great to add such evaluation as an example to `llama.cpp` and add instructions for...

Follow up on the idea by @ikawrakow in https://github.com/ggerganov/llama.cpp/pull/729#issuecomment-1521825435 ### Q5_0 ```c #define QK5_0 32 typedef struct { ggml_fp16_t d; // delta uint8_t qh[4]; // 5-th bit of quants (uint32_t)...

ggml : spread compute across threads in chunks

5

This is an alternative way of distributing the work across workers. Not sure yet if it is more efficient. The idea is for each thread to process small chunk of...

WIP IN PROGRESS - hparams: https://github.com/facebookresearch/segment-anything/blob/efeab7296ab579d4a261e554eca80faf6b33924a/segment_anything/build_sam.py#L13-L44 - cmd: ``` python scripts/amg.py --checkpoint ./sam_vit_b_01ec64.pth --model-type vit_b --input img.jpg --output img.out --device cpu ``` PTH tensors for ViT-B ``` image_encoder.neck.0.weight torch.Size([256, 768,...

Make this compatible with [nn.Linear](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html) Needed for [SAM](https://github.com/ggerganov/ggml/pull/74) inference

ggml : add macros for accessing tensors to reduce code duplication

7

The following pattern is repeated extensively throughout `ggml.c`: ```c const int64_t ne00 = src0->ne[0]; const int64_t ne01 = src0->ne[1]; const int64_t ne02 = src0->ne[2]; const int64_t ne03 = src0->ne[3]; const...

ggml : improve API to allow allocating compute graphs on the heap

5

Currently, `ggml` forces the user to allocate the compute graphs on the stack. The `ggml` API should be extended to support using heap allocated graphs.

ggml : improve memory management

3

One of the biggest problems with `ggml` currently is that the user needs to manually pre-calculate the necessary sizes for all the `ggml_context` objects that they create. This is a...

See https://github.com/ggerganov/ggml/pull/224

ggml : improve CI + add more tests

9

The current state of the testing framework is pretty bad - we have a few simple test tools in [tests](https://github.com/ggerganov/ggml/tree/master/tests), but these are not maintained properly and are quite rudimentary....