serving
serving copied to clipboard
tensorflow
Poor documentation regarding hosting models on e.g., AWS s3 bucket.
Feature Request
Describe the problem the feature is intended to solve
Hi, I have been using TensorFlow Serving for over a year now and I just today stumbled across One can host models on a remote storage e.g., a bucket on AWS s3.
How come this is not documented on the official serving configuration page? Moreover, are the models downloaded from the remote storage locally to the TensorFlow Serving container assuming One is running the server using Docker?
The reason I am asking the latter question is to investigate if hosting models on a remote storage impacts the model latency.
Describe the solution
A thorough description of ways to host models to serve on remote storage on the TensorFlow Serving documentation page.
- It's good that TF serving can accept remotely hosted models. However, if it's not documented, I don't think we can rely on the assumption that remote models would continue to be supported. TF serving might stop supporting it without public announcement.
- I don't think remote models would affect the latency. It might slow down the first loading of the model, but once it's loaded into the memory, It wouldn't affect the latency whether the source was locally or remotely hosted.
- It's good that TF serving can accept remotely hosted models. However, if it's not documented, I don't think we can rely on the assumption that remote models would continue to be supported. TF serving might stop supporting it without public announcement.
This will lock people to use older supported versions, and also break lot of pipelines if you are directly pulling the latest tensorflow-serving docker image
- It's good that TF serving can accept remotely hosted models. However, if it's not documented, I don't think we can rely on the assumption that remote models would continue to be supported. TF serving might stop supporting it without public announcement.
This will lock people to use older supported versions, and also break lot of pipelines if you are directly pulling the latest tensorflow-serving docker image
So its actually happening, the docker image for the latest tag is using 2.5.1 , which was released 8 months ago https://hub.docker.com/layers/tensorflow/serving/latest/images/sha256-6651f4839e1124dbde75ee531825112af0a6b8ef082c88ab14ca53eb69a2e4bb?context=explore and has s3 support.
But the Docker image with tag 2.7.0 doesn't have s3 support
Logs for reference
servable en with error UNIMPLEMENTED: File system scheme 's3' not implemented
This is an expected behaviour since version 2.6 https://github.com/tensorflow/tensorflow/issues/51583
I'm using S3 hosting for my models as well. This regression of feature means I will be unable to move to a newer version of tfserving due to the way I have to test and validate my models prior to putting them into production. This is also not a feature provided by SIG IO/TF-IO at this time. I understand and respect not wanting this feature to be part of the core project. It makes more sense to have this functionality in an optional library for sure. Can we get some consideration on having this functionality being placed in an optional library instead of being removed wholesale?
@hampusrosvall,
It's good that TF serving can accept remotely hosted models. However, if it's not documented, I don't think we can rely on the assumption that remote models would continue to be supported. TF serving might stop supporting it without public announcement.
As mentioned above, since remote models are not supported there is no official documentation from TF serving. Users stumbled across this issue can refer to this documentation to load model from s3 storage.
@johnbarney, To answer your question, Cloud filesystem implementation moved to tensorflow-io. From TF serving 2.7 onwards, users will have to install tensorflow-io and import it to load model from S3 storage. Reference
Hope this answers your question. Thank you!
Thanks for the reply. Last I checked the issue was tensorflow-io wasn’t loaded by tfserving which meant I couldn’t load models from S3. I could only load the tensorflow-io afterward to load the dataset. This may have been solved by now but we solved it internally using wrappers, Docker and AWS magic. It makes sense to have this functionality in a library, but the library didn’t cover all the use cases unfortunately.
On Fri, Nov 25, 2022 at 5:51 PM, singhniraj08 @.***> wrote:
@.***(https://github.com/hampusrosvall),
It's good that TF serving can accept remotely hosted models. However, if it's not documented, I don't think we can rely on the assumption that remote models would continue to be supported. TF serving might stop supporting it without public announcement.
As mentioned above, since remote models are not supported there is no official documentation from TF serving. Users stumbled across this issue can refer to this documentation to load model from s3 storage.
@.***(https://github.com/johnbarney), To answer your question, Cloud filesystem implementation moved to tensorflow-io. From TF serving 2.7 onwards, users will have to install tensorflow-io and import it to load model from S3 storage. Reference
Hope this answers your question. Thank you!
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>
Closing this due to inactivity. Please take a look into the answers provided above, feel free to reopen and post your comments(if you still have queries on this). Thank you!