how to use lightseq inference engine

[lyzKF]

hello, I have 1 question about how to use lightseq inference engine

I trained an en2de model based on fairseq, which is a variant of the transformer, i.e. I modified the FFN layer. Can I use the lightseq inference engine to speed up the model? If not, what should I do

looking forward for your reply, thank you!

[zjersey]

For now, lightseq inference cannot support structure modification. You can describe your network and we will evaluate if it can be supported.

[lyzKF]

@zjersey yep, get it. if i modify it by myself, i should modify the model.proto, encoder.cc.cu, decoder.cc.cu and some kernel function files, then i compile source codes. is that. right?

[hexisyztem]

Yes, but there are a lot of details that can be confusing for third-party developers.

lyzKF

[lyzKF]

@zjersey @hexisyztem yep, get it, thank you.👍👍👍

[hexisyztem]

Compile environment you can refer to this: https://github.com/bytedance/lightseq/blob/master/docker/Tritonserver/Dockerfile.

[hexisyztem]

If you have any questions, you can ask me and I will try my best to help. In the future, we will consider providing users with fine-grained operators to facilitate self-assembly of models

[lyzKF]

@hexisyztem thank you👍，i will email you, if i was confusing the details about "lightseq" inference engine.