Timon Käch

Is there a possibilty to offload the model to ram?

5

Hello! I really want to test out the 7b model. Is there any option to offload it to ram? My GPU is a rtx 3070ti with 8gb vram and I...

Long time until generation starts when using big context

10

When just saying like "Hello, who are you?", I get like 200ms/token and it starts generating almost instantly. On the other hand, when I paste a small text (e.g. search...

Speed when splitting model across 2 gpu's

3

Hello Guys! I wan't to buy 2x3060 12gb as I get them for only 500.- Then I want to use llama 30b 4bit with it which uses 20gb gpu memory,...

Issue with emoji decoding

1

When the model wants to output an emoji, this error comes up: `Debugging middleware caught exception in streamed response at a point where response headers were already sent. Traceback (most...

API returns only none, none

1

### Describe the bug When using the api-example-stream.py I only get "None None" in the output. When I remove to get only the data of the request I get this:...

API doesn't work.

7

### Describe the bug I can't get the api to work. Either it says ConnectionRefused or when you change the port to 7860 it tells me some strange html errors....

Quality small vs large model

3

Has anyone already compared the quality of the small models vs the big models? I'm quite interested in the difference. If other people would also like to have more info...

Stable Diffusion SD XL support

7

Hello! It would be super cool to accelerate the stable diffusion xl models, as they are pretty slow because of the 1024x1024 res. I think it could work pretty easily,...

Hello @rvaidun Thank you for this amazing project. It worked for a very long time. I got a new laptop and I can't login there. When I enter my swiss...

SDXL Support

3

Hello Is SDXL support planned, as SDXL is slow on most computers? Kind regards, Timon Käch