OlivierDehaene

This backend is an example of how you would go about adding other backends to TEI. It is currently out-dated but could be updated to support AMD.

It seems its linked to an issue with Hyper: https://github.com/hyperium/hyper/issues/1790 #161 solves the issue by using another memory allocator.

Do you have a graph of the memory increase? And if you have v1.0.0 vs v1.1.0 that would be amazing.

Ok I will keep this in my prio list but it must be very deep in the stack and might take some time to find. > The worker/client program in...

@novak2000 can you use 1.1 and keep the memory resources limit? I'm wondering if the container will be killed or not on 1.1.

Ok thanks for this info. I'm more or less off this week so I will keep digging when I find the time.

It should be ok to add. Candle and pytorch are not Iso in features but it should have everything to support it. You wont be able to use flash attention...

Well the model needs to be ported from torch to candle which, depending on the model can take from one hour up to a day. We do not have an...

T5 models like instructor are not yet supported.

What is the status of this PR?