Nick Hill

A new bug was introduced in 0.4.2, but fixed in https://github.com/vllm-project/vllm/pull/4737. Please try with that PR or as a workaround you can also install `tensorizer`. This should resolve such errors...

Thanks @lamoghaighoroje!

We've now opened a PR to add support for this #3466

I have tested this with my own code/usecase but wanted to check that there is interest in the contribution before also updating any applicable unit tests. I also wonder whether...

Thanks @gante ... I'm kind of new to this but will figure out how to verify/update the tests per your request. The main problem I've run into though is newly-failing...

Thanks @gante, it sounds like you followed a similar path to me w.r.t. trying different arrangements of the logic to get around this. I was guessing this couldn't be the...

Thanks @michaelbenayoun. You are right that this seems to be the fact that a symbolic proxy tensor is introduced somewhere, however I think that this stems from the tensor-based indexing...

Thanks @michaelbenayoun .. the `len` problem can be avoided by adding `torch.fx.wrap('len')`, which I'd done in the prior commit but removed in this latest commit since it seemed futile (just...

@gante @michaelbenayoun I've got torch.fx to work with the changes now by using `torch.gather` instead of tensor based indexing and adding a couple of new tensor methods to the metadata...

Thanks @gante, glad I was able to contribute. Thank you for your fast responses and for all the great work you and team do.