No module named 'cloudpickle.cloudpickle_fast' when launching JOB
/kind bug
What steps did you take and what happened: I launched a simple JOB made of a basic function (just print a statement)
from kubeflow import fairing
from kubeflow.fairing import TrainJob
from kubeflow.fairing.backends import KubeflowAWSBackend
AWS_ACCOUNT_ID = fairing.cloud.aws.guess_account_id()
AWS_REGION = 'us-east-1'
DOCKER_REGISTRY = '{}.dkr.ecr.{}.amazonaws.com'.format(AWS_ACCOUNT_ID, AWS_REGION)
def simple_function():
print("Hello Fairing world")
job = TrainJob(simple_function,base_docker_image='AWS_ECR_REPO/python:3.8-slim', docker_registry=DOCKER_REGISTRY, backend=KubeflowAWSBackend())
job.submit()
After launching the submit command the container is built correctly and eventually pushed into the registry, but once launched I get the following error
[W 200210 15:07:40 manager:255] Waiting for fairing-job-55nc4-rccmv to start...
[E 200210 15:07:41 manager:272] Failed to launch fairing-job-55nc4-rccmv, reason: Error, message: None
[W 200210 15:07:42 job:162] Cleaning up job fairing-job-55nc4...
Traceback (most recent call last):
File "/app/function_shim.py", line 2, in <module>
import cloudpickle
File "/app/cloudpickle/__init__.py", line 9, in <module>
from cloudpickle.cloudpickle_fast import CloudPickler, dumps, dump
ModuleNotFoundError: No module named 'cloudpickle.cloudpickle_fast'
What did you expect to happen: Job correctly launched
Anything else you would like to add:
If I launched the container manually (docker run ...) I get the same error
Environment:
- Fairing version: 0.7.0
- Kubeflow version: (version number can be found at the bottom left corner of the Kubeflow dashboard): 0.7.0
- Minikube version: NA
- Kubernetes version: (use
kubectl version): 1.14 - OS (e.g. from
/etc/os-release): Amazon Linux
Issue-Label Bot is automatically applying the labels:
| Label | Probability |
|---|---|
| kind/bug | 0.99 |
Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! Links: app homepage, dashboard and code for this bot.
Hi @Luke035 , i got the same problem, are there any solutions for this ?. Thanks