[Feature Request]: Compile model w/ NEO to target AWS Lambda

[ecurtin2]

Hey all, thanks for the nice software!

I've been struggling to find a good way to deploy a sagemaker model in a serverless fashion.

It seems like a great solution would be to compile a trained model using NEO to target the AWS lambda runtime. Perhaps even this could be added to a lambda layer.

Thus the serverless deployment of a sagemaker model would be to simply compile -> include a layer in a lambda function and invoke that lambda using whatever techniques are appropriate for the problem.

Is this a thing that's in the pipeline? I tried to do it myself but I had a tough time figuring out how the various components work.

[mvsusp]

Hi @ecurtin2 ,

Although your suggestion would result in a very nice feature, we do not have any immediate plans to work on it.

Please, let me know if you have additional questions.

Thanks for using SageMaker!

[marcelpanse]

Would love this!

[9hgg]

This would be amazing!

[j-adamczyk]

+1, this would be great. Additionally, support for SageMaker Serverless should be added in the similar fashion.

[j-adamczyk]

Any news on this?

[mohanasudhan]

@ecurtin2 @j-adamczyk @mvsusp @jorisgu SageMaker support for Serverless inference launched last year https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints.html

Please reopen the issue if you see any issue in using the feature.

[j-adamczyk]

@mohanasudhan why has this been closed? How can we use Neo to target either Lambda (preferably) or SageMaker Serverless Inference (it kinda also works)? Neither one is listed in the documentation as a supported instance type: https://docs.aws.amazon.com/sagemaker/latest/dg/neo-supported-cloud.html. Or should we use it as a TargetDevice, as listed here? https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_OutputConfig.html. In that case, just using lambda will work? Why is this not stated anywhere in the docs then? Or how can we use Neo with SageMaker Serverless Inference then? How is this feature request completed?

[mohanasudhan]

Apologies for not making it clear. Neo is supported in SageMaker and just noticed that serverless target isn't supported today. Will keep this issue open and will reach to respective team for correspondence.

[j-adamczyk]

Ok, so why is there lambda listed as a target? Is that a mistake? I get that Neo is a part of SageMaker, that is obvious, but I would very much prefer to use Lambda for deploying my functions. If that won't be a supported target, only SageMaker Serverless, I will write my own optimization & compilation code.

[rishrayc]

Amazon SageMaker Neo does support Lambda as a target but it's on a deprecation path. So we do not recommend using the feature. Unfortunately, support for Serverless inference as a target in Neo is currently not prioritized. So we cannot share a firm date for the feature support. Please reply back if you have any questions/suggestions.

[ShwetaSingh801]

Hi @ecurtin2,

Thanks for using SageMaker and taking the time to suggest ways to improve SageMaker Python SDK. We have added your feature request to our backlog of feature requests and may consider putting it into future SDK versions. I will go ahead and close the issue now, please let me know if you have any more feedback. Let me know if you have any other questions.

Best, Shweta