NOFRAUD Latam comments

Repositories
Issues
Comments

Results 4 comments of


                                            NOFRAUD Latam

Ollama is not using the 100% of RTX4000 VRAM (18 of 20GB)

So, is it a bug in the VRAM memory allocation or is it the expected behavior ?

Ollama is not using the 100% of RTX4000 VRAM (18 of 20GB)

Thank you @orlyandico, I was able to increase by 2 the number of layers deployed in the VRAM GPU using Mixtral 3BitQ (from 29/33 to 31/33). I gained some performance...

Added support for specifying an arbitrary GBNF compatible grammar

This would be awesome, particularly when you need a specific output like "yes or no".

The typing animation is working?

I made a little modification to this script in order to set automatic scroll-down in the message area just after the dot animation (the original script fails to do that...