Casper

Hi @chu-tianxiang, I added an issue to track it. I attempted to put a device guard in place and it fixes the illegal memory access error, but then results in...

@lroberts7 it seems your tests are failing for reasons unrelated to this PR. I think you may have an environment issue or some problem with the GPUs.

> The PR breaks the mixtral unit test previously and I pushed a fix for it, but I'm still seeing `illegal memory access` in the [CI test](https://buildkite.com/vllm/ci/builds/1578#018dd49d-ec7f-4013-b9ab-7962be808b99) after the commit....

It must be a type-hint bug somewhere. If you correctly type hint your code, you will not get that error. The problem is that the `ccxt.Exchange` type works as an...

Looking forward to this one, it will enable this library to more accurately account for fees, especially with the commission tuple!

@kernc when do you expect to merge this into the main branch and create a new release?

> I tested implementing this but ran into some issues that I'm not really sure why they are happening. I can perhaps make a draft PR. That would be great!...

@PanQiWei Old issue but would love to see some updated support for models like MPT and Falcon. MPT keeps pumping out new models, we got [30B-8k](https://huggingface.co/mosaicml/mpt-30b) and [7B-8k](https://huggingface.co/mosaicml/mpt-7b-8k) now, so...

> > @PanQiWei Old issue but would love to see some updated support for models like MPT and Falcon. > > MPT keeps pumping out new models, we got [30B-8k](https://huggingface.co/mosaicml/mpt-30b)...

> As a result of the current experiment with LLaMA -7B, an exact comparison cannot be made, but the approximate results are as follows. > > [LLaMA-7B](https://arxiv.org/abs/2302.13971) BPW Wikitext2 >...