sagemaker-python-sdk icon indicating copy to clipboard operation
sagemaker-python-sdk copied to clipboard

Published 20 hours ago • verified publisher icon aws

feat: Add transformers 4.29 - PyTorch 2.0.0 training and inference

Open phall1 opened this issue 1 year ago • 34 comments

Issue #, if available: #3896

Description of changes: Update Image URI for:

Transformers 4.29 - PyTorch 2.0 Inference DLC Transformers 4.29 - PyTorch 2.0 Training DLC

Testing done:

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

  • [x] I have read the CONTRIBUTING doc
  • [ ] I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
  • [ ] I used the commit message format described in CONTRIBUTING
  • [ ] I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
  • [ ] I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

  • [ ] I have added tests that prove my fix is effective or that my feature works (if appropriate)
  • [ ] I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
  • [ ] I have checked that my tests are not configured for a specific region or account (if appropriate)
  • [ ] I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

phall1 avatar Jun 06 '23 02:06 phall1

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: 9be32b33ba94954095396094a4476e0c7f3343fb
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot avatar Jun 06 '23 06:06 sagemaker-bot

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: 9be32b33ba94954095396094a4476e0c7f3343fb
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot avatar Jun 06 '23 06:06 sagemaker-bot

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-slow-tests
  • Commit ID: 9be32b33ba94954095396094a4476e0c7f3343fb
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot avatar Jun 06 '23 06:06 sagemaker-bot

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: 9be32b33ba94954095396094a4476e0c7f3343fb
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot avatar Jun 06 '23 06:06 sagemaker-bot

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: 9be32b33ba94954095396094a4476e0c7f3343fb
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot avatar Jun 06 '23 07:06 sagemaker-bot

Codecov Report

Attention: 32 lines in your changes are missing coverage. Please review.

Comparison is base (3057f21) 90.27% compared to head (a494586) 90.23%. Report is 64 commits behind head on master.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #3904      +/-   ##
==========================================
- Coverage   90.27%   90.23%   -0.04%     
==========================================
  Files         993      316     -677     
  Lines       87861    29282   -58579     
==========================================
- Hits        79313    26424   -52889     
+ Misses       8548     2858    -5690
Files Coverage Δ
src/sagemaker/base_predictor.py 90.90% <100.00%> (ø)
src/sagemaker/config/config_schema.py 100.00% <100.00%> (ø)
src/sagemaker/estimator.py 92.88% <ø> (ø)
src/sagemaker/feature_store/feature_group.py 78.82% <ø> (ø)
...emaker/feature_store/feature_processor/__init__.py 100.00% <ø> (ø)
...eature_store/feature_processor/_config_uploader.py 97.43% <100.00%> (ø)
...er/feature_store/feature_processor/_data_source.py 100.00% <100.00%> (ø)
...ore/feature_processor/_feature_processor_config.py 100.00% <100.00%> (ø)
.../feature_store/feature_processor/_params_loader.py 100.00% <ø> (ø)
...ker/feature_store/feature_processor/_validation.py 84.05% <100.00%> (ø)
... and 32 more

... and 1267 files with indirect coverage changes

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov-commenter avatar Jun 06 '23 07:06 codecov-commenter

Updated python version for torch 2.0 compatibility per the failed tests in the failed builds

phall1 avatar Jun 06 '23 07:06 phall1

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: a84574b9d6ee71b80ea7cb3799ab7c32e67f246d
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot avatar Jun 07 '23 05:06 sagemaker-bot

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: a84574b9d6ee71b80ea7cb3799ab7c32e67f246d
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot avatar Jun 07 '23 05:06 sagemaker-bot

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: a84574b9d6ee71b80ea7cb3799ab7c32e67f246d
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot avatar Jun 07 '23 05:06 sagemaker-bot

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-slow-tests
  • Commit ID: a84574b9d6ee71b80ea7cb3799ab7c32e67f246d
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot avatar Jun 07 '23 05:06 sagemaker-bot

Looks like the image isn't available yet, per here:

https://github.com/aws/deep-learning-containers/blob/master/available_images.md

Do we know if this is being worked on or how I could contribute to getting this image available? Error from logs:

sagemaker.exceptions.UnexpectedStatusException: Error for Processing job test-huggingface-2023-06-07-04-47-23-957: Failed. Reason: ClientError: API error (404): manifest for 763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-pytorch-training:2.0.0-transformers4.29.0-gpu-py310-cu118-ubuntu20.04 not found: manifest unknown: Requested image not found

phall1 avatar Jun 07 '23 05:06 phall1

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: a84574b9d6ee71b80ea7cb3799ab7c32e67f246d
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot avatar Jun 07 '23 05:06 sagemaker-bot

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: aeb5cc3baa7651f4d8bfae3a0fbe9b1fbbdff356
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot avatar Jun 07 '23 17:06 sagemaker-bot

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: aeb5cc3baa7651f4d8bfae3a0fbe9b1fbbdff356
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot avatar Jun 07 '23 17:06 sagemaker-bot

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: aeb5cc3baa7651f4d8bfae3a0fbe9b1fbbdff356
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot avatar Jun 07 '23 17:06 sagemaker-bot

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: aeb5cc3baa7651f4d8bfae3a0fbe9b1fbbdff356
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot avatar Jun 07 '23 17:06 sagemaker-bot

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-slow-tests
  • Commit ID: aeb5cc3baa7651f4d8bfae3a0fbe9b1fbbdff356
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot avatar Jun 07 '23 20:06 sagemaker-bot

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: 2b1464f6ca77958580d722e5cf4d8f1dac2974e8
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot avatar Jun 16 '23 17:06 sagemaker-bot

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: 2b1464f6ca77958580d722e5cf4d8f1dac2974e8
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot avatar Jun 16 '23 17:06 sagemaker-bot

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-slow-tests
  • Commit ID: 2b1464f6ca77958580d722e5cf4d8f1dac2974e8
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot avatar Jun 16 '23 17:06 sagemaker-bot

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-unit-tests
  • Commit ID: 2b1464f6ca77958580d722e5cf4d8f1dac2974e8
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot avatar Jun 16 '23 18:06 sagemaker-bot

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: 2b1464f6ca77958580d722e5cf4d8f1dac2974e8
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot avatar Jun 16 '23 19:06 sagemaker-bot

Looks like the image isn't available yet, per here:

https://github.com/aws/deep-learning-containers/blob/master/available_images.md

Do we know if this is being worked on or how I could contribute to getting this image available?

Error from logs:


sagemaker.exceptions.UnexpectedStatusException: Error for Processing job test-huggingface-2023-06-07-04-47-23-957: Failed. Reason: ClientError: API error (404): manifest for 763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-pytorch-training:2.0.0-transformers4.29.0-gpu-py310-cu118-ubuntu20.04 not found: manifest unknown: Requested image not found

Same error as before - bumping my question l

phall1 avatar Jun 16 '23 19:06 phall1

Looks like the image isn't available yet, per here: aws/deep-learning-containers@master/available_images.md Do we know if this is being worked on or how I could contribute to getting this image available? Error from logs:


sagemaker.exceptions.UnexpectedStatusException: Error for Processing job test-huggingface-2023-06-07-04-47-23-957: Failed. Reason: ClientError: API error (404): manifest for 763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-pytorch-training:2.0.0-transformers4.29.0-gpu-py310-cu118-ubuntu20.04 not found: manifest unknown: Requested image not found

Same error as before - bumping my question l

We will have to check with DLC team to verify on the image availability. @ShiboXing Can you help answer this question?

knikure avatar Jun 16 '23 21:06 knikure

@phall1 Can you check with someone in DLC if transformers 4.29 is supported with PT 2.0. If not, can we close this PR?

knikure avatar Aug 14 '23 21:08 knikure

It would be great to get some feedback about this, as we would like to add support for further transformers version (e.g. https://github.com/aws/sagemaker-python-sdk/pull/4076)

oleksiishevchuk avatar Aug 23 '23 18:08 oleksiishevchuk

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-local-mode-tests
  • Commit ID: a49458696abc1a875a4bc1c06cb606d08f3a2752
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot avatar Oct 26 '23 01:10 sagemaker-bot

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-pr
  • Commit ID: a49458696abc1a875a4bc1c06cb606d08f3a2752
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot avatar Oct 26 '23 01:10 sagemaker-bot

AWS CodeBuild CI Report

  • CodeBuild project: sagemaker-python-sdk-notebook-tests
  • Commit ID: a49458696abc1a875a4bc1c06cb606d08f3a2752
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot avatar Oct 26 '23 01:10 sagemaker-bot