Not defining a global provider.runtime breaks dockerizePip: true

[ToneVDB]

Are you certain it's a bug?

• [X] Yes, it looks like a bug

Are you using the latest plugin release?

• [X] Yes, I'm using the latest plugin release

Is there an existing issue for this?

• [X] I have searched existing issues, it hasn't been reported yet

Issue description

We have a project that uses different runtimes per function: function1: docker function2: python3.9

Because of this, in the provider section, we did not define a runtime setting, as we did this on an individual function level. (package.individually was set to true) When deploying this stack it caused no issue at all, untill we needed to set the dockerizePip to true

Doing this caused the following error to appear:

× Stack application-dev failed to deploy (2s)
Environment: linux, node 18.16.1, framework 3.33.0 (local), plugin 6.2.3, SDK 4.3.2
Credentials: Local, environment variables
Docs:        docs.serverless.com
Support:     forum.serverless.com
Bugs:        github.com/serverless/serverless/issues

Error:
Running "docker run --rm -v /home/vsts/.cache/serverless-python-requirements/312994c1479362d239d9d5bf26421488a6a04688248779b8d3d3ec9ec8d32241_x86_64_slspyc:/var/task:z -v /home/vsts/.cache/serverless-python-requirements/downloadCacheslspyc:/var/useDownloadCache:z public.ecr.aws/sam/build-undefined:latest-x86_64 /bin/sh -c chown -R 0\:0 /var/useDownloadCache && python -m pip install -t /var/task/ -r /var/task/requirements.txt --cache-dir /var/useDownloadCache && chown -R 1001\:122 /var/task && chown -R 1001\:122 /var/useDownloadCache" failed with: "Unable to find image 'public.ecr.aws/sam/build-undefined:latest-x86_64' locally
docker: Error response from daemon: repository public.ecr.aws/sam/build-undefined not found: name unknown: The repository with name 'build-undefined' does not exist in the registry with id 'sam'.
See 'docker run --help'."
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Tracing back the error caused me to look where public.ecr.aws/sam/build- was used in the repository. It led me here As in our usecase serverless.service.provider.runtime was not set on a project lvl, it caused the code to replace this with the string undefined.

This image ofcourse does not exist, causing the docker command to crash out. Setting the provider.runtime string to python3.9 fixes this issue.

Service configuration (serverless.yml) content

service: application

frameworkVersion: '3'

package:
  individually: true
  patterns:
    - '!./**/**'
    - .requirements.zip

provider:
  name: aws
  stage: ${sls:stage}
  timeout: 900 
  memorySize: 2048

  # if you do not define it here, it breaks the dockerizePip: true
  # remove the comment on the below line to fix the error
  # runtime: python3.9

  environment:
    ENV: ${sls:stage}
    DEPLOYED: TRUE

  # Lambda needs access to S3, secretsmanager, dynamodb and logs
  iam:
    role:
      statements:
        - Effect: Allow
          Action:
            - "s3:*"
          Resource: "*"

        - Effect: Allow
          Action:
            - "secretsmanager:GetSecretValue"
          Resource: "*"

        - Effect: Allow
          Action:
            - "dynamodb:*"
          Resource: "*"

        - Effect: Allow
          Action:
            - "logs:*"
          Resource: "*"

  # Serverless stores our docker image of the app in ECR
  ecr:
    images:
      appimage_scrape:
        path: ./
        file: scrape.dockerfile

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    usePoetry: true
    dockerizePip: true
    requirePoetryLockFile: true
    zip: true
    slim: true
    strip: false
    

functions:
  ma_scrape:
    environment:
      ENV: ${sls:stage}
    image:
      name: appimage_scrape
    events:
      # Invoke Lambda function every day at 6:15h UTC from Mon-Fri
      - schedule: cron(15 6 ? * MON-FRI *)

  ma_clean:
    handler: src.handler.run_clean
    runtime: python3.9
    environment:
      ENV: ${sls:stage}
    events:
      - s3:
          bucket: ${self:service}-${sls:stage}-databucket
          event: s3:ObjectCreated:*
          rules:
            - prefix: raw/
          existing: true
          forceDeploy: true
    package:
      patterns:
        - src/application/clean/**
        - src/application/common/**
        - src/handler.py

  ma_score:
    handler: src.handler.run_score
    runtime: python3.9
    environment:
      ENV: ${sls:stage}
    events:
      - s3:
          bucket: ${self:service}-${sls:stage}-databucket
          event: s3:ObjectCreated:*
          rules:
            - prefix: clean/
            - suffix: info.json
          existing: true
          forceDeploy: true
    package:
      patterns:
        - src/application/score/**
        - src/application/common/**
        - src/handler.py

  ma_match:
    handler: src.handler.run_match
    runtime: python3.9
    environment:
      ENV: ${sls:stage}
    events:
      # Invoke Lambda function every day at 6:45h UTC from Mon-Fri
      - schedule: cron(45 6 ? * MON-FRI *)
    
    package:
      patterns:
        - src/application/match/**
        - src/application/common/**
        - src/handler.py

  ma_send:
    handler: src.handler.run_send
    runtime: python3.9
    environment:
      ENV: ${sls:stage}
    events:
      # Invoke Lambda function every day at 7h UTC from Mon-Fri
      - schedule: cron(0 7 ? * MON-FRI *)
    
    package:
      patterns:
        - src/application/send/**
        - src/application/common/**
        - src/handler.py

Command name and used flags

serverless deploy --verbose --region eu-west-1 --stage dev

Command output

$ /home/vsts/work/1/s/node_modules/.bin/serverless deploy --verbose --region eu-west-1 --stage dev

Deploying application to stage dev (eu-west-1)

Adding Python requirements helper to .
Generating requirements.txt from poetry.lock
Parsed requirements.txt from pyproject.toml in /home/vsts/work/1/s/.serverless/requirements.txt
Installing requirements from "/home/vsts/.cache/serverless-python-requirements/312994c1479362d239d9d5bf26421488a6a04688248779b8d3d3ec9ec8d32241_x86_64_slspyc/requirements.txt"
Docker Image: public.ecr.aws/sam/build-undefined:latest-x86_64
Using download cache directory /home/vsts/.cache/serverless-python-requirements/downloadCacheslspyc
Running docker run --rm -v /home/vsts/.cache/serverless-python-requirements/312994c1479362d239d9d5bf26421488a6a04688248779b8d3d3ec9ec8d32241_x86_64_slspyc\:/var/task\:z -v /home/vsts/.cache/serverless-python-requirements/downloadCacheslspyc\:/var/useDownloadCache\:z public.ecr.aws/sam/build-undefined\:latest-x86_64 /bin/sh -c 'chown -R 0\\:0 /var/useDownloadCache && python -m pip install -t /var/task/ -r /var/task/requirements.txt --cache-dir /var/useDownloadCache && chown -R 1001\\:122 /var/task && chown -R 1001\\:122 /var/useDownloadCache'...

× Stack application-dev failed to deploy (2s)
Environment: linux, node 18.16.1, framework 3.33.0 (local), plugin 6.2.3, SDK 4.3.2
Credentials: Local, environment variables
Docs:        docs.serverless.com
Support:     forum.serverless.com
Bugs:        github.com/serverless/serverless/issues

Error:
Running "docker run --rm -v /home/vsts/.cache/serverless-python-requirements/312994c1479362d239d9d5bf26421488a6a04688248779b8d3d3ec9ec8d32241_x86_64_slspyc:/var/task:z -v /home/vsts/.cache/serverless-python-requirements/downloadCacheslspyc:/var/useDownloadCache:z public.ecr.aws/sam/build-undefined:latest-x86_64 /bin/sh -c chown -R 0\:0 /var/useDownloadCache && python -m pip install -t /var/task/ -r /var/task/requirements.txt --cache-dir /var/useDownloadCache && chown -R 1001\:122 /var/task && chown -R 1001\:122 /var/useDownloadCache" failed with: "Unable to find image 'public.ecr.aws/sam/build-undefined:latest-x86_64' locally
docker: Error response from daemon: repository public.ecr.aws/sam/build-undefined not found: name unknown: The repository with name 'build-undefined' does not exist in the registry with id 'sam'.
See 'docker run --help'."
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
yarn run v1.22.19
warning package.json: No license field

Environment information

Framework Core: 3.33.0 (local)
Plugin: 6.2.3
SDK: 4.3.2


serverless-python-requirements: 6.0.0

Solution/ workaround

Under the provider section, define your runtime (like python 3,9)

[lockstock123]

for those like me that assume that no comments means no solution, the OP has given the solution (thankyou):

provider:
   runtime: python3.9

could this issue be specific to Python 3.9?

[ToneVDB]

First of all, I am sorry that the solution was not that clear, I'll edit the issue. Second, I am not sure, but if you turn off dockerizePip, it stops failing. That leads me to belive that this breaks for all python versions, and not just 3.9. (as pip is used in all of the other python versions as well ofc.)

[kalanchoej]

Also broken with 3.11. I was defining the runtime version per-function.

[duncanista]

Similar to me – if I didn't specify a provider, it would fail. Even if the functions had a specific runtime.

[duncanista]

I ran into this again – for my surprise, I forgot about it and just found that I previously commented. Still think this should be on the function level, rather the provider level.

[pgrzesik]

Hey @duncanista 👋 I totally missed this issue the first time around. What version of the plugin are you using right now? I've observed some weird behavior with dockerizePip image resolution and want to pin down this is also present for the latest version. Also, just to make sure I understand the issue well, this only happens when runtime is missing on provider-level, correct?

[duncanista]

Hey @pgrzesik,

I'm using the latest version of both the plugin and the framework.

And yes, when you don't provide a runtime at the provider level, dockerizePip crashes. I actually would expect to use the specific function runtime, in case I want to use multiple runtimes for different features.

What happens specifically is that the public image to be grabbed is found as undefined, making it to try to pull the build-undefined, instead of build-{runtime}.

Let me know if you need more information!