Working with a saved image in S3 appears to be broken

[3mei]

Hitting the following two errors when specifying a tarball, as per https://scar.readthedocs.io/en/latest/advanced_usage.html

1. The first one is 'S3Properties' object has no attribute 'storage_path_input' and appears to be thrown because an input_bucket needs to be specified in addition to the deployment_bucket in the yaml file

2. If specifying both the deployment_bucket and the input_bucket, the following is thrown:

Error: failed to extract container: /tmp/udocker_image.tar.gz
Error: loading failed
Command '['python3', '/tmp/tmp8wznq7pc/udocker/udocker.py', 'load', '-i', '/tmp/udocker_image.tar.gz']' returned non-zero exit status 1.

Here is a log from scar.log:

2019-12-18 20:41:15,364 - root - INFO - Using existent 'faas-supervisor' layer
2019-12-18 20:41:16,629 - root - ERROR - Command '['python3', '/tmp/tmp8wznq7pc/udocker/udocker.py', 'load', '-i', '/tmp/udocker_image.tar.gz']' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/home/scar/.local/lib/python3.7/site-packages/scar/scarcli.py", line 84, in main
    ScarCLI().parse_arguments()
  File "/home/scar/.local/lib/python3.7/site-packages/scar/exceptions.py", line 32, in wrapper
    return func(*args, **kwargs)
  File "/home/scar/.local/lib/python3.7/site-packages/scar/scarcli.py", line 79, in parse_arguments
    merged_args['scar']['func']()
  File "/home/scar/.local/lib/python3.7/site-packages/scar/scarcli.py", line 36, in init
    self.cloud_provider.init()
  File "/home/scar/.local/lib/python3.7/site-packages/scar/exceptions.py", line 32, in wrapper
    return func(*args, **kwargs)
  File "/home/scar/.local/lib/python3.7/site-packages/scar/providers/aws/controller.py", line 98, in init
    self._create_lambda_function()
  File "/home/scar/.local/lib/python3.7/site-packages/scar/exceptions.py", line 32, in wrapper
    return func(*args, **kwargs)
  File "/home/scar/.local/lib/python3.7/site-packages/scar/providers/aws/controller.py", line 216, in _create_lambda_function
    response = self.aws_lambda.create_function()
  File "/home/scar/.local/lib/python3.7/site-packages/scar/exceptions.py", line 32, in wrapper
    return func(*args, **kwargs)
  File "/home/scar/.local/lib/python3.7/site-packages/scar/providers/aws/lambdafunction.py", line 91, in create_function
    self._set_function_code()
  File "/home/scar/.local/lib/python3.7/site-packages/scar/exceptions.py", line 32, in wrapper
    return func(*args, **kwargs)
  File "/home/scar/.local/lib/python3.7/site-packages/scar/providers/aws/lambdafunction.py", line 178, in _set_function_code
    FunctionPackager(self.aws, self.supervisor_version).create_zip()
  File "/home/scar/.local/lib/python3.7/site-packages/scar/exceptions.py", line 32, in wrapper
    return func(*args, **kwargs)
  File "/home/scar/.local/lib/python3.7/site-packages/scar/providers/aws/functioncode.py", line 74, in create_zip
    self._manage_udocker_images()
  File "/home/scar/.local/lib/python3.7/site-packages/scar/providers/aws/functioncode.py", line 94, in _manage_udocker_images
    self.udocker.prepare_udocker_image()
  File "/home/scar/.local/lib/python3.7/site-packages/scar/providers/aws/udocker.py", line 98, in prepare_udocker_image
    cli_msg="Loading image file")
  File "/home/scar/.local/lib/python3.7/site-packages/scar/utils.py", line 80, in execute_command_with_msg
    cmd_out = subprocess.check_output(command, cwd=cmd_wd).decode('utf-8')
  File "/usr/local/lib/python3.7/subprocess.py", line 395, in check_output
    **kwargs).stdout
  File "/usr/local/lib/python3.7/subprocess.py", line 487, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['python3', '/tmp/tmp8wznq7pc/udocker/udocker.py', 'load', '-i', '/tmp/udocker_image.tar.gz']' returned non-zero exit status 1.

[alpegon]

Hi, thanks for using scar ! We are working in a new function definition language and we are currently updating scar and all the related libraries. The error that you are mentioning is fixed in this new version. If you want to try the beta version of scar with the error fixed, you can check the fdl branch. If you prefer to wait, we expect to launch a new version very soon. Regards

[3mei]

Hey @alpegon, thanks for creating SCAR and for the pointer! I'll take a look at the fdl branch

[3mei]

@alpegon It looks like the fdl branch is also throwing an error. I was wondering if there is working example of pulling an image from S3 instead of docker hub.

[srisco]

Hello @3mei, thanks for trying the fdl branch, I just released SCAR 4.0.0 with support for Function Definition Language and multiple bugfixes, including some related with your issue.

About pulling images from S3, currently private container images can only be attached to the function's deployment package. You can do it adding the image_file variable, for example:

functions:
  aws:
  - lambda:
      name: scar-cowsay
      container:
        image_file: minicow.tar.gz

The problem with this approach is that deployment package (with the image file) is uploaded directly in the function payload, that cannot exceed 50MB. To define functions with bigger deployment packages, AWS Lambda allows the upload to S3. To do it with SCAR, you have to add the variable deployment and bucket with the name of the S3 bucket where you want to upload the deployment package. Example:

functions:
  aws:
  - lambda:
      name: scar-cowsay
      container:
        image_file: cowsay.tar.gz
      deployment:
        bucket: scar-test

Remember that, in any case, deployment packages have a limit of 250MB (unzipped, including layers).

Cheers.