Derek Kozikowski

@dbarbuzzi , good catch. @horheynm, it may be good to understand why pipeline.py overrides batch_size when engine.py is already doing it differently? and if the statement on the old lines...

Another open question to resolve w/ this code is the handling of spaces in tokens. here's an image from my debugger showing that huggingface generated text does not have spaces...

> Thanks @derekk-nm, I think the new test is looking good! > > > ``` > > # now repeat using two gpus > > # specifically doing it here,...

@robertgshaw2-neuralmagic , @dbarbuzzi , @dhuangnm , I've pushed a bunch of changes to this WIP PR. Key things are specified in the commit message. the things that remain are: *...

I believe this latest commit correctly decodes the token ids from the HFRunnerNM, but I can't find a way to decode the token ids in the logprobs keys using _decode_token_by_position_index....

[server_basic_correctness_mistral_results_202404301351.txt](https://github.com/neuralmagic/nm-vllm/files/15170475/server_basic_correctness_mistral_results_202404301351.txt) this attachment is the latest output from the test execution. There are two tests using the same model, one test using a single gpu, and another with 2 (via...

Thank you @robertgshaw2-neuralmagic for the consultation. I've updated the script to use the existing comparison function, and corrected the method that decodes the logprob keys. The two tests are now...

This test is failing today. Something's been broken over the weekend. The exception is: ``` ==== server startup command args ==== --model mistralai/Mistral-7B-Instruct-v0.2 --max-model-len 4096 --disable-log-requests --tensor-parallel-size 2 --dtype half...

I don't understand why the build was skipped. I didn't try to skip it.

After rebasing this branch onto main, the test is passing for me with the single Mistral model: ``` /root/pyvenv/nmv1/bin/python3 -m pytest --forked --import-mode importlib tests/basic_correctness/test_basic_server_correctness.py -k test_models_on_server ============================= test session...