[Question] TF Serving Support

[terrykong]

Hi,

I was curious if seamless integration with TF Serving is on the roadmap?

I see that there is support for remote models here remote_model.py, but is there anything beyond that we can look forward to?

We usually have a model server running that serves our models, so it would be great to reuse the models and resources allocated to the model server.

Thanks!

[iftenney]

It's on the roadmap to have more "seamless" integration with more frameworks, including TensorFlow Serving, but no ETA yet.

That being said: for a specific model, if you have a Python class that can call the model server, it should be fairly easy to map the flat dict format of PredictRequest/PredictResponse onto LIT types (see https://github.com/PAIR-code/lit/blob/main/docs/python_api.md#models).