mmdeploy
mmdeploy copied to clipboard
open-mmlab
[Feature Request] AWS Neuron backend
Describe the feature
I'd like to request adding AWS Neuron as a supported backend for deployment to AWS Inferentia chips.
Motivation
Inferentia chips are a new, very fast and cost effective cloud deployment option.
Related resources
PyTorch recently published a blog post describing how to use the Neuron SDK to export HuggingFace models for Inferentia serving: https://pytorch.org/blog/amazon-ads-case-study/
Here's a list of currently supported PyTorch operators: https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/neuron-cc-ops/neuron-cc-ops-pytorch.html
Hi, @austinmw sorry for replying late. We are interested in supporting more platforms and devices. Let us take some time to do research about AWS Neuron and I will get back to you asap.
Hi, @austinmw I am afraid we won't be able to support AWS Neuron this year. The following devices are in our H2 TO-DO list:
- macOS M1
- RISC-V
- Huawei Atlas
@lvhan028 Thanks for your reply. M1/M2 chip deployment is exciting!
Maybe can circle back to this in 2023? I think Inferentia chip adoption and workloads will increase in a very noticeable way by then.
@lvhan028 Thanks for your reply. M1/M2 chip deployment is exciting!
Maybe can circle back to this in 2023? I think Inferentia chip adoption and workloads will increase in a very noticeable way by then.
@austinmw Do you have any data to prove that? If so, could you share it with me so that I can report it to my boss.
@lvhan028 Thanks, I think a lot more information about this specifically will come from the upcoming AWS Silicon Innovation Day event.
Here are some currently published customer success stories. There are a few more major customer testimonials that will be made public in the next few weeks.
Also just a note on feasibility, I've been told that since you already have a TorchScript backend, the road to Neuron support is actually very short.