Issues with running serve.sh

[palebluedot19]

I installed Nanoflow by following the instructions in the installation section. The installation was done locally (not docker container). After the installation is done, I run the command ./serve.sh and get an error:

Enter the number corresponding to your model choice: 5
Traceback (most recent call last):
  File "/home/ubuntu/Nanoflow/pipeline/utils/serve_8B.py", line 6, in <module>
    import pllm_python
ModuleNotFoundError: No module named 'pllm_python'
cat: trace.req_words: No such file or directory

To fix this error, I tried to run the following command:

export PYTHONPATH=${PYTHONPATH}:/path/to/Nanoflow/pipeline/build

This doesn't help.

I created a pllm_python directory in site-packages

mkdir pllm_python

and copied all the libraries in the pipeline/build directory to pllm_python in site-packages I got around the above error, but ran into another error

Traceback (most recent call last):
  File "/home/ubuntu/Nanoflow/pipeline/utils/serve_8B.py", line 17, in <module>
    from pybindUtil import toGPU, toGPUShard, initUpdateData, genInitData, load_pipeline_config
  File "/home/ubuntu/Nanoflow/pipeline/utils/pybindUtil.py", line 91, in <module>
    prefill_tokens_num: int) -> list[pllm_python.VortexUpdateData]:
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pllm_python' has no attribute 'VortexUpdateData'

How do I get it to work? Thanks.

[Wazrrr]

We have changed our codebase from c++ to python, now it shoule be easier to launch the program.