Ying Xiong

You can checkout the release tag to run the examples. Master may be not compatible with pypi latest version.

You can try `pip3 install -v -e ./` to check if the build is successful.

No, the weights are saved in fp32. 1e-4 precision loss in fp32 mode is acceptable. You try a higher level metric to measure the difference. ________________________________ 发件人: SamuelSiu ***@***.***> 发送时间:...

Sampling doesn't have multiple outputs. You can feed a batch of the same inputs to get multiple outputs. For the wrong results, you can compile with debug info to find...

For your own model, I suggest compiling with debug info and checking each layer's output. If you can reproduce the wrong result on an open model, I can find out...

I think it may relate to tokenizer bos and eos config, you can check if they are passed to model_config correctly.

We haven't dived too much into this. When implementing lightseq, we try to use as few parameters and few shared intermediate variables as possible, and a noticeable GPU memory reduction...

`pip install lightseq` should be inference by fp16. On Fri, Apr 8, 2022 at 6:08 PM Jhin ***@***.***> wrote: > 我想问下lightseq内部是不是默认就是用的fp16做的推理呀，感觉数据不太对，能否开放源码直接编译inference，通过pip install > -e .的形式把inference也做进去，方便调试。 > > — > Reply...

That's right On Tue, Apr 12, 2022 at 2:01 PM Jhin ***@***.***> wrote: > What do you mean? Is it the default fp16? I see your demo that comparison >...

You can check `docs/inference/build.md` to build inference from source.