fairing icon indicating copy to clipboard operation
fairing copied to clipboard

No module named 'cloudpickle.cloudpickle_fast' when launching JOB

Open Luke035 opened this issue 5 years ago • 2 comments

/kind bug

What steps did you take and what happened: I launched a simple JOB made of a basic function (just print a statement)

from kubeflow import fairing
from kubeflow.fairing import TrainJob
from kubeflow.fairing.backends import KubeflowAWSBackend

AWS_ACCOUNT_ID = fairing.cloud.aws.guess_account_id()
AWS_REGION = 'us-east-1'
DOCKER_REGISTRY = '{}.dkr.ecr.{}.amazonaws.com'.format(AWS_ACCOUNT_ID, AWS_REGION)

def simple_function():
    print("Hello Fairing world")

job = TrainJob(simple_function,base_docker_image='AWS_ECR_REPO/python:3.8-slim', docker_registry=DOCKER_REGISTRY, backend=KubeflowAWSBackend())

job.submit()

After launching the submit command the container is built correctly and eventually pushed into the registry, but once launched I get the following error

[W 200210 15:07:40 manager:255] Waiting for fairing-job-55nc4-rccmv to start...
[E 200210 15:07:41 manager:272] Failed to launch fairing-job-55nc4-rccmv, reason: Error, message: None
[W 200210 15:07:42 job:162] Cleaning up job fairing-job-55nc4...
Traceback (most recent call last):
  File "/app/function_shim.py", line 2, in <module>
    import cloudpickle
  File "/app/cloudpickle/__init__.py", line 9, in <module>
    from cloudpickle.cloudpickle_fast import CloudPickler, dumps, dump
ModuleNotFoundError: No module named 'cloudpickle.cloudpickle_fast'

What did you expect to happen: Job correctly launched

Anything else you would like to add: If I launched the container manually (docker run ...) I get the same error

Environment:

  • Fairing version: 0.7.0
  • Kubeflow version: (version number can be found at the bottom left corner of the Kubeflow dashboard): 0.7.0
  • Minikube version: NA
  • Kubernetes version: (use kubectl version): 1.14
  • OS (e.g. from /etc/os-release): Amazon Linux

Luke035 avatar Feb 10 '20 16:02 Luke035

Issue-Label Bot is automatically applying the labels:

Label Probability
kind/bug 0.99

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! Links: app homepage, dashboard and code for this bot.

issue-label-bot[bot] avatar Feb 10 '20 16:02 issue-label-bot[bot]

Hi @Luke035 , i got the same problem, are there any solutions for this ?. Thanks

MinhManPham avatar Nov 22 '22 12:11 MinhManPham