NicolasHug
pip installation with Scikit-surprise in AWS sam Error
Description
I am trying to deploy a customer recommendation API on AWS Lambda using the sam CLI and docker following this tutorial: https://sejalv.medium.com/containerized-ml-deployment-with-aws-lambda-680540fb92f4
I am using a base python image from aws's public ecr and I've added scikit-surprise and numpy in my requirements.txt file as stated in your installation doc but when I run $ sam build I get the following error message:
Using legacy setup.py install for scikit-surprise, since package 'wheel' is not installed.
Installing collected packages: pytz, six, python-dateutil, numpy, pandas, joblib, scipy, scikit-surprise, urllib3, chardet, certifi, idna, requests, threadpoolctl, scikit-learn
Running setup.py install for scikit-surprise: started
Running setup.py install for scikit-surprise: finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /var/lang/bin/python3.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-6gksgbo5/scikit-surprise/setup.py'"'"'; __file__='"'"'/tmp/pip-install-6gksgbo5/scikit-surprise/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-xrkjvq3d/install-record.txt --single-version-externally-managed --compile --install-headers /var/lang/include/python3.7m/scikit-surprise
cwd: /tmp/pip-install-6gksgbo5/scikit-surprise/
Complete output (53 lines):
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/surprise
copying surprise/dump.py -> build/lib.linux-x86_64-3.7/surprise
copying surprise/builtin_datasets.py -> build/lib.linux-x86_64-3.7/surprise
copying surprise/accuracy.py -> build/lib.linux-x86_64-3.7/surprise
copying surprise/dataset.py -> build/lib.linux-x86_64-3.7/surprise
copying surprise/reader.py -> build/lib.linux-x86_64-3.7/surprise
copying surprise/__main__.py -> build/lib.linux-x86_64-3.7/surprise
copying surprise/trainset.py -> build/lib.linux-x86_64-3.7/surprise
copying surprise/__init__.py -> build/lib.linux-x86_64-3.7/surprise
copying surprise/utils.py -> build/lib.linux-x86_64-3.7/surprise
creating build/lib.linux-x86_64-3.7/surprise/model_selection
copying surprise/model_selection/split.py -> build/lib.linux-x86_64-3.7/surprise/model_selection
copying surprise/model_selection/validation.py -> build/lib.linux-x86_64-3.7/surprise/model_selection
copying surprise/model_selection/__init__.py -> build/lib.linux-x86_64-3.7/surprise/model_selection
copying surprise/model_selection/search.py -> build/lib.linux-x86_64-3.7/surprise/model_selection
creating build/lib.linux-x86_64-3.7/surprise/prediction_algorithms
copying surprise/prediction_algorithms/algo_base.py -> build/lib.linux-x86_64-3.7/surprise/prediction_algorithms
copying surprise/prediction_algorithms/knns.py -> build/lib.linux-x86_64-3.7/surprise/prediction_algorithms
copying surprise/prediction_algorithms/predictions.py -> build/lib.linux-x86_64-3.7/surprise/prediction_algorithms
copying surprise/prediction_algorithms/baseline_only.py -> build/lib.linux-x86_64-3.7/surprise/prediction_algorithms
copying surprise/prediction_algorithms/__init__.py -> build/lib.linux-x86_64-3.7/surprise/prediction_algorithms
copying surprise/prediction_algorithms/random_pred.py -> build/lib.linux-x86_64-3.7/surprise/prediction_algorithms
running egg_info
writing scikit_surprise.egg-info/PKG-INFO
writing dependency_links to scikit_surprise.egg-info/dependency_links.txt
writing entry points to scikit_surprise.egg-info/entry_points.txt
writing requirements to scikit_surprise.egg-info/requires.txt
writing top-level names to scikit_surprise.egg-info/top_level.txt
reading manifest file 'scikit_surprise.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'scikit_surprise.egg-info/SOURCES.txt'
copying surprise/similarities.c -> build/lib.linux-x86_64-3.7/surprise
copying surprise/similarities.pyx -> build/lib.linux-x86_64-3.7/surprise
copying surprise/prediction_algorithms/co_clustering.c -> build/lib.linux-x86_64-3.7/surprise/prediction_algorithms
copying surprise/prediction_algorithms/co_clustering.pyx -> build/lib.linux-x86_64-3.7/surprise/prediction_algorithms
copying surprise/prediction_algorithms/matrix_factorization.c -> build/lib.linux-x86_64-3.7/surprise/prediction_algorithms
copying surprise/prediction_algorithms/matrix_factorization.pyx -> build/lib.linux-x86_64-3.7/surprise/prediction_algorithms
copying surprise/prediction_algorithms/optimize_baselines.c -> build/lib.linux-x86_64-3.7/surprise/prediction_algorithms
copying surprise/prediction_algorithms/optimize_baselines.pyx -> build/lib.linux-x86_64-3.7/surprise/prediction_algorithms
copying surprise/prediction_algorithms/slope_one.c -> build/lib.linux-x86_64-3.7/surprise/prediction_algorithms
copying surprise/prediction_algorithms/slope_one.pyx -> build/lib.linux-x86_64-3.7/surprise/prediction_algorithms
running build_ext
building 'surprise.similarities' extension
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/surprise
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/var/lang/lib/python3.7/site-packages/numpy/core/include -I/var/lang/include/python3.7m -c surprise/similarities.c -o build/temp.linux-x86_64-3.7/surprise/similarities.o
unable to execute 'gcc': No such file or directory
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /var/lang/bin/python3.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-6gksgbo5/scikit-surprise/setup.py'"'"'; __file__='"'"'/tmp/pip-install-6gksgbo5/scikit-surprise/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-xrkjvq3d/install-record.txt --single-version-externally-managed --compile --install-headers /var/lang/include/python3.7m/scikit-surprise Check the logs for full command output.
WARNING: You are using pip version 20.1.1; however, version 21.1.2 is available.
You should consider upgrading via the '/var/lang/bin/python3.7 -m pip install --upgrade pip' command.
Build Failed
Error: HelloWorldFunction failed to build: The command '/bin/sh -c pip install -r requirements.txt' returned a non-zero code: 1
Steps/Code to Reproduce
You need an aws account and aws-cli installed
Find a directory you want to work from and make sure docker is running and you're logged in:
pip install aws-sam-cli
sam init
select options 1 (template) then 2 (base image) then 5 (python3.7) - this will create a project template folder in this directory
add scikit-surprise and numpy to requirements.txt file and then run
sam build
Expected Results
I tried adding wheel to the requirements.txt file but it's still not working.
Please let me know if there's any other information you need from me
Actual Results
Versions
Darwin-20.3.0-x86_64-i386-64bit public.ecr.aws/lambda/python:3.7 as base - python 3.7 ecr base image surprise 1.1.1
Hi @thabied , sorry I'm really not familiar with AWS but to install surprise you can either install a binary from conda, OR you'll have to build from source (e.g. when installing with pip install). That will require a compiler, Cython, etc (see our building from source instructions)
Hi @thabied , sorry I'm really not familiar with AWS but to install surprise you can either install a binary from conda, OR you'll have to build from source (e.g. when installing with pip install). That will require a compiler, Cython, etc (see our building from source instructions)
Hi @thabied, I had the same issue and installing gcc on my Docker image that I use with AWS sam worked fine for me. I added this line to my Dockerfile before installing Surprise:
RUN yum install gcc -y
I am deploying my Lambda function as a container image. See here for an example.
