kubeflow-kale
Parser must be a string or character stream, not dict
While using Kale in AKS cluster, iam always getting following error while compiling and uploading the pipeline.
If anyone has solution to this issue , kindly let me know.
Kale version: 0.5.0 AKS Version: v1.19.7 Kubeflow Version: 1.0.0
Here is the log details:
2021-03-01 11:45:37 run:83 [[DEBUG]] [TID=d458eymim5] [] Decoding ctx of RPC function 'rok.check_rok_availability'
2021-03-01 11:45:37 run:95 [[DEBUG]] [TID=d458eymim5] [] Decoding kwargs of RPC function 'rok.check_rok_availability'
2021-03-01 11:45:37 run:104 [[DEBUG]] [TID=d458eymim5] [] Importing RPC function 'rok.check_rok_availability'
2021-03-01 11:45:40 run:114 [[INFO]] [TID=d458eymim5] [] Executing RPC function 'check_rok_availability()'
2021-03-01 11:45:40 rok:123 [[ERROR]] [TID=d458eymim5] [] Failed to initialize RokClient
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 117, in check_rok_availability
rok = _get_client()
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 42, in _get_client
_client = RokClient()
File "build/bdist.linux-x86_64/egg/rok_gw_client/client.py", line 175, in init
ValueError: A Rok API URL must be provided either via the url' argument or the ROK_GW_URL' environment variable.
2021-03-01 11:45:40 run:120 [[ERROR]] [TID=d458eymim5] [] RPC function 'check_rok_availability' raised an RPCError
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 117, in check_rok_availability
rok = _get_client()
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 42, in _get_client
_client = RokClient()
File "build/bdist.linux-x86_64/egg/rok_gw_client/client.py", line 175, in init
ValueError: A Rok API URL must be provided either via the url' argument or the ROK_GW_URL' environment variable.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run
result = func(request, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 126, in check_rok_availability
trans_id=request.trans_id)
kale.rpc.errors.RPCServiceUnavailableError
2021-03-01 11:46:03 run:83 [[DEBUG]] [TID=utv8q8mksy] [] Decoding ctx of RPC function 'nb.resume_notebook_path'
2021-03-01 11:46:03 run:95 [[DEBUG]] [TID=utv8q8mksy] [] Decoding kwargs of RPC function 'nb.resume_notebook_path'
2021-03-01 11:46:03 run:104 [[DEBUG]] [TID=utv8q8mksy] [] Importing RPC function 'nb.resume_notebook_path'
2021-03-01 11:46:04 run:114 [[INFO]] [TID=utv8q8mksy] [] Executing RPC function 'resume_notebook_path(server_root=/home/jovyan)'
2021-03-01 11:47:45 run:83 [[DEBUG]] [TID=h6zg5ogl3c] [] Decoding ctx of RPC function 'nb.explore_notebook'
2021-03-01 11:47:45 run:95 [[DEBUG]] [TID=h6zg5ogl3c] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.explore_notebook'
2021-03-01 11:47:45 run:104 [[DEBUG]] [TID=h6zg5ogl3c] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.explore_notebook'
2021-03-01 11:47:45 run:114 [[INFO]] [TID=h6zg5ogl3c] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'explore_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb)'
2021-03-01 11:47:45 run:83 [[DEBUG]] [TID=759gi3qsbb] [] Decoding ctx of RPC function 'nb.get_base_image'
2021-03-01 11:47:45 run:95 [[DEBUG]] [TID=759gi3qsbb] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.get_base_image'
2021-03-01 11:47:45 run:104 [[DEBUG]] [TID=759gi3qsbb] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.get_base_image'
2021-03-01 11:47:45 run:114 [[INFO]] [TID=759gi3qsbb] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'get_base_image()'
2021-03-01 11:47:45 run:83 [[DEBUG]] [TID=eaxdwp5at4] [] Decoding ctx of RPC function 'nb.find_poddefault_labels_on_server'
2021-03-01 11:47:45 run:95 [[DEBUG]] [TID=eaxdwp5at4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.find_poddefault_labels_on_server'
2021-03-01 11:47:45 run:104 [[DEBUG]] [TID=eaxdwp5at4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.find_poddefault_labels_on_server'
2021-03-01 11:47:45 run:114 [[INFO]] [TID=eaxdwp5at4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'find_poddefault_labels_on_server()'
2021-03-01 11:47:45 nb:205 [[INFO]] [TID=eaxdwp5at4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Retrieving PodDefaults applied to server...
2021-03-01 11:47:45 nb:211 [[INFO]] [TID=eaxdwp5at4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Retrieved applied PodDefaults: []
2021-03-01 11:47:45 nb:215 [[INFO]] [TID=eaxdwp5at4] [/home/jovyan/Simple Notebook Pipeline.ipynb] PodDefault labels applied on server:
2021-03-01 11:47:46 run:83 [[DEBUG]] [TID=rekhin8drk] [] Decoding ctx of RPC function 'kfp.list_experiments'
2021-03-01 11:47:46 run:95 [[DEBUG]] [TID=rekhin8drk] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'kfp.list_experiments'
2021-03-01 11:47:46 run:104 [[DEBUG]] [TID=rekhin8drk] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'kfp.list_experiments'
2021-03-01 11:47:46 run:114 [[INFO]] [TID=rekhin8drk] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'list_experiments()'
2021-03-01 11:47:46 run:125 [[ERROR]] [TID=rekhin8drk] [/home/jovyan/Simple Notebook Pipeline.ipynb] RPC function 'list_experiments' raised an unhandled exception
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run
result = func(request, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 34, in list_experiments
c = _get_client()
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 27, in _get_client
_client = kfp.Client()
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 119, in init
self._configure_incluster_auth()
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 727, in _configure_incluster_auth
self.set_user_namespace(old_ns_setting or _get_pod_namespace())
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 239, in set_user_namespace
with open(Client.LOCAL_KFP_CONTEXT, 'w') as f:
PermissionError: [Errno 13] Permission denied: '/home/jovyan/.config/kfp/context.json'
2021-03-01 11:48:11 run:83 [[DEBUG]] [TID=d6shlr2egc] [] Decoding ctx of RPC function 'nb.validate_notebook'
2021-03-01 11:48:11 run:95 [[DEBUG]] [TID=d6shlr2egc] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook'
2021-03-01 11:48:11 run:104 [[DEBUG]] [TID=d6shlr2egc] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.validate_notebook'
2021-03-01 11:48:11 run:114 [[INFO]] [TID=d6shlr2egc] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'experiment_name': 'calculation exppriments', 'katib_run': False, 'pipeline_description': 'this is test pipeline', 'pipeline_name': 'newpipeline', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': [], 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}})'
2021-03-01 11:48:11 run:83 [[DEBUG]] [TID=ul8skouxpt] [] Decoding ctx of RPC function 'nb.compile_notebook'
2021-03-01 11:48:11 run:95 [[DEBUG]] [TID=ul8skouxpt] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.compile_notebook'
2021-03-01 11:48:11 run:104 [[DEBUG]] [TID=ul8skouxpt] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.compile_notebook'
2021-03-01 11:48:11 run:114 [[INFO]] [TID=ul8skouxpt] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'compile_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'experiment_name': 'calculation exppriments', 'katib_run': False, 'pipeline_description': 'this is test pipeline', 'pipeline_name': 'newpipeline', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': [], 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}}, debug=False)'
2021-03-01 11:48:11 core:146 [[DEBUG]] [TID=ul8skouxpt] [/home/jovyan/Simple Notebook Pipeline.ipynb] ------------- Kale Start Run -------------
2021-03-01 11:48:11 core:205 [[INFO]] [TID=ul8skouxpt] [/home/jovyan/Simple Notebook Pipeline.ipynb] Pipeline code saved at /home/jovyan/newpipeline-w8cw0.kale.py
2021-03-01 11:48:12 run:83 [[DEBUG]] [TID=clr8x31dd9] [] Decoding ctx of RPC function 'kfp.upload_pipeline'
2021-03-01 11:48:12 run:95 [[DEBUG]] [TID=clr8x31dd9] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'kfp.upload_pipeline'
2021-03-01 11:48:12 run:104 [[DEBUG]] [TID=clr8x31dd9] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'kfp.upload_pipeline'
2021-03-01 11:48:12 run:114 [[INFO]] [TID=clr8x31dd9] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'upload_pipeline(pipeline_package_path=newpipeline-w8cw0.pipeline.yaml, pipeline_metadata={'experiment_name': 'calculation exppriments', 'pipeline_name': 'newpipeline-w8cw0', 'pipeline_description': 'this is test pipeline', 'pipeline_args': '', 'pipeline_args_names': '', 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'volumes': [], 'abs_working_dir': '/home/jovyan', 'marshal_volume': True, 'marshal_path': '/marshal', 'snapshot_volumes': False, 'autosnapshot': False, 'steps_defaults': [], 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'volume_access_mode': 'rwm', 'parameters_names': [], 'parameters_types': [], 'parameters_values': []}, overwrite=False)'
2021-03-01 11:48:12 run:125 [[ERROR]] [TID=clr8x31dd9] [/home/jovyan/Simple Notebook Pipeline.ipynb] RPC function 'upload_pipeline' raised an unhandled exception
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run
result = func(request, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 93, in upload_pipeline
client = _get_client()
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 27, in _get_client
_client = kfp.Client()
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 119, in init
self._configure_incluster_auth()
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 727, in _configure_incluster_auth
self.set_user_namespace(old_ns_setting or _get_pod_namespace())
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 239, in set_user_namespace
with open(Client.LOCAL_KFP_CONTEXT, 'w') as f:
PermissionError: [Errno 13] Permission denied: '/home/jovyan/.config/kfp/context.json'
2021-03-01 11:48:32 run:83 [[DEBUG]] [TID=1zmzvqqz8a] [] Decoding ctx of RPC function 'nb.validate_notebook'
2021-03-01 11:48:32 run:95 [[DEBUG]] [TID=1zmzvqqz8a] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook'
2021-03-01 11:48:32 run:104 [[DEBUG]] [TID=1zmzvqqz8a] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.validate_notebook'
2021-03-01 11:48:32 run:114 [[INFO]] [TID=1zmzvqqz8a] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'experiment_name': 'calculation exppriments', 'katib_run': False, 'pipeline_description': 'this is test pipeline', 'pipeline_name': 'newpipeline', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': [], 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}})'
2021-03-01 11:48:32 run:83 [[DEBUG]] [TID=buox86rnmq] [] Decoding ctx of RPC function 'nb.compile_notebook'
2021-03-01 11:48:32 run:95 [[DEBUG]] [TID=buox86rnmq] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.compile_notebook'
2021-03-01 11:48:32 run:104 [[DEBUG]] [TID=buox86rnmq] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.compile_notebook'
2021-03-01 11:48:32 run:114 [[INFO]] [TID=buox86rnmq] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'compile_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'experiment_name': 'calculation exppriments', 'katib_run': False, 'pipeline_description': 'this is test pipeline', 'pipeline_name': 'newpipeline', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': [], 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}}, debug=False)'
2021-03-01 11:48:32 core:146 [[DEBUG]] [TID=buox86rnmq] [/home/jovyan/Simple Notebook Pipeline.ipynb] ------------- Kale Start Run -------------
2021-03-01 11:48:32 core:205 [[INFO]] [TID=buox86rnmq] [/home/jovyan/Simple Notebook Pipeline.ipynb] Pipeline code saved at /home/jovyan/newpipeline-9y29i.kale.py
2021-03-01 11:48:32 run:83 [[DEBUG]] [TID=9m6azmqhtp] [] Decoding ctx of RPC function 'kfp.upload_pipeline'
2021-03-01 11:48:32 run:95 [[DEBUG]] [TID=9m6azmqhtp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'kfp.upload_pipeline'
2021-03-01 11:48:32 run:104 [[DEBUG]] [TID=9m6azmqhtp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'kfp.upload_pipeline'
2021-03-01 11:48:32 run:114 [[INFO]] [TID=9m6azmqhtp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'upload_pipeline(pipeline_package_path=newpipeline-9y29i.pipeline.yaml, pipeline_metadata={'experiment_name': 'calculation exppriments', 'pipeline_name': 'newpipeline-9y29i', 'pipeline_description': 'this is test pipeline', 'pipeline_args': '', 'pipeline_args_names': '', 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'volumes': [], 'abs_working_dir': '/home/jovyan', 'marshal_volume': True, 'marshal_path': '/marshal', 'snapshot_volumes': False, 'autosnapshot': False, 'steps_defaults': [], 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'volume_access_mode': 'rwm', 'parameters_names': [], 'parameters_types': [], 'parameters_values': []}, overwrite=False)'
2021-03-01 11:48:32 run:125 [[ERROR]] [TID=9m6azmqhtp] [/home/jovyan/Simple Notebook Pipeline.ipynb] RPC function 'upload_pipeline' raised an unhandled exception
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run
result = func(request, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 93, in upload_pipeline
client = _get_client()
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 27, in _get_client
_client = kfp.Client()
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 119, in init
self._configure_incluster_auth()
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 727, in _configure_incluster_auth
self.set_user_namespace(old_ns_setting or _get_pod_namespace())
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 239, in set_user_namespace
with open(Client.LOCAL_KFP_CONTEXT, 'w') as f:
PermissionError: [Errno 13] Permission denied: '/home/jovyan/.config/kfp/context.json'
2021-03-01 11:48:59 run:83 [[DEBUG]] [TID=6gh062igyc] [] Decoding ctx of RPC function 'nb.validate_notebook'
2021-03-01 11:48:59 run:95 [[DEBUG]] [TID=6gh062igyc] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook'
2021-03-01 11:48:59 run:104 [[DEBUG]] [TID=6gh062igyc] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.validate_notebook'
2021-03-01 11:48:59 run:114 [[INFO]] [TID=6gh062igyc] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'experiment_name': 'calculation exppriments', 'katib_run': False, 'pipeline_description': 'this is test pipeline', 'pipeline_name': 'newpipeline', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': [], 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}})'
2021-03-01 11:48:59 run:83 [[DEBUG]] [TID=jz1pmv0rtp] [] Decoding ctx of RPC function 'nb.compile_notebook'
2021-03-01 11:48:59 run:95 [[DEBUG]] [TID=jz1pmv0rtp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.compile_notebook'
2021-03-01 11:48:59 run:104 [[DEBUG]] [TID=jz1pmv0rtp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.compile_notebook'
2021-03-01 11:48:59 run:114 [[INFO]] [TID=jz1pmv0rtp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'compile_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'experiment_name': 'calculation exppriments', 'katib_run': False, 'pipeline_description': 'this is test pipeline', 'pipeline_name': 'newpipeline', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': [], 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}}, debug=False)'
2021-03-01 11:48:59 core:146 [[DEBUG]] [TID=jz1pmv0rtp] [/home/jovyan/Simple Notebook Pipeline.ipynb] ------------- Kale Start Run -------------
2021-03-01 11:49:00 core:205 [[INFO]] [TID=jz1pmv0rtp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Pipeline code saved at /home/jovyan/newpipeline-4p5xw.kale.py
2021-03-01 11:49:08 run:83 [[DEBUG]] [TID=uglyl7e4k1] [] Decoding ctx of RPC function 'nb.validate_notebook'
2021-03-01 11:49:08 run:95 [[DEBUG]] [TID=uglyl7e4k1] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook'
2021-03-01 11:49:08 run:104 [[DEBUG]] [TID=uglyl7e4k1] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.validate_notebook'
2021-03-01 11:49:08 run:114 [[INFO]] [TID=uglyl7e4k1] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'experiment_name': 'calculation exppriments', 'katib_run': False, 'pipeline_description': 'this is test pipeline', 'pipeline_name': 'newpipeline', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': [], 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}})'
2021-03-01 11:49:08 run:83 [[DEBUG]] [TID=qsz52aeqlg] [] Decoding ctx of RPC function 'nb.compile_notebook'
2021-03-01 11:49:08 run:95 [[DEBUG]] [TID=qsz52aeqlg] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.compile_notebook'
2021-03-01 11:49:08 run:104 [[DEBUG]] [TID=qsz52aeqlg] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.compile_notebook'
2021-03-01 11:49:08 run:114 [[INFO]] [TID=qsz52aeqlg] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'compile_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'experiment_name': 'calculation exppriments', 'katib_run': False, 'pipeline_description': 'this is test pipeline', 'pipeline_name': 'newpipeline', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': [], 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}}, debug=False)'
2021-03-01 11:49:08 core:146 [[DEBUG]] [TID=qsz52aeqlg] [/home/jovyan/Simple Notebook Pipeline.ipynb] ------------- Kale Start Run -------------
2021-03-01 11:49:09 core:205 [[INFO]] [TID=qsz52aeqlg] [/home/jovyan/Simple Notebook Pipeline.ipynb] Pipeline code saved at /home/jovyan/newpipeline-a7dfv.kale.py
2021-03-01 11:49:09 run:83 [[DEBUG]] [TID=lfe9ig9mgb] [] Decoding ctx of RPC function 'kfp.upload_pipeline'
2021-03-01 11:49:09 run:95 [[DEBUG]] [TID=lfe9ig9mgb] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'kfp.upload_pipeline'
2021-03-01 11:49:09 run:104 [[DEBUG]] [TID=lfe9ig9mgb] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'kfp.upload_pipeline'
2021-03-01 11:49:09 run:114 [[INFO]] [TID=lfe9ig9mgb] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'upload_pipeline(pipeline_package_path=newpipeline-a7dfv.pipeline.yaml, pipeline_metadata={'experiment_name': 'calculation exppriments', 'pipeline_name': 'newpipeline-a7dfv', 'pipeline_description': 'this is test pipeline', 'pipeline_args': '', 'pipeline_args_names': '', 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'volumes': [], 'abs_working_dir': '/home/jovyan', 'marshal_volume': True, 'marshal_path': '/marshal', 'snapshot_volumes': False, 'autosnapshot': False, 'steps_defaults': [], 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'volume_access_mode': 'rwm', 'parameters_names': [], 'parameters_types': [], 'parameters_values': []}, overwrite=False)'
2021-03-01 11:49:09 run:125 [[ERROR]] [TID=lfe9ig9mgb] [/home/jovyan/Simple Notebook Pipeline.ipynb] RPC function 'upload_pipeline' raised an unhandled exception
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run
result = func(request, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 93, in upload_pipeline
client = _get_client()
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 27, in _get_client
_client = kfp.Client()
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 119, in init
self._configure_incluster_auth()
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 727, in _configure_incluster_auth
self.set_user_namespace(old_ns_setting or _get_pod_namespace())
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 239, in set_user_namespace
with open(Client.LOCAL_KFP_CONTEXT, 'w') as f:
PermissionError: [Errno 13] Permission denied: '/home/jovyan/.config/kfp/context.json'
2021-03-01 11:55:37 run:83 [[DEBUG]] [TID=q5dvo05iyu] [] Decoding ctx of RPC function 'nb.validate_notebook'
2021-03-01 11:55:37 run:95 [[DEBUG]] [TID=q5dvo05iyu] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook'
2021-03-01 11:55:37 run:104 [[DEBUG]] [TID=q5dvo05iyu] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.validate_notebook'
2021-03-01 11:55:37 run:114 [[INFO]] [TID=q5dvo05iyu] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'experiment_name': 'calculation exppriments', 'katib_run': False, 'pipeline_description': 'this is test pipeline', 'pipeline_name': 'newpipeline', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': [], 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}})'
2021-03-01 11:55:37 run:83 [[DEBUG]] [TID=fwibookgb4] [] Decoding ctx of RPC function 'nb.compile_notebook'
2021-03-01 11:55:37 run:95 [[DEBUG]] [TID=fwibookgb4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.compile_notebook'
2021-03-01 11:55:37 run:104 [[DEBUG]] [TID=fwibookgb4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.compile_notebook'
2021-03-01 11:55:37 run:114 [[INFO]] [TID=fwibookgb4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'compile_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'experiment_name': 'calculation exppriments', 'katib_run': False, 'pipeline_description': 'this is test pipeline', 'pipeline_name': 'newpipeline', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': [], 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}}, debug=False)'
2021-03-01 11:55:37 core:146 [[DEBUG]] [TID=fwibookgb4] [/home/jovyan/Simple Notebook Pipeline.ipynb] ------------- Kale Start Run -------------
2021-03-01 11:55:37 core:205 [[INFO]] [TID=fwibookgb4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Pipeline code saved at /home/jovyan/newpipeline-tko28.kale.py
2021-03-01 11:55:38 run:83 [[DEBUG]] [TID=q0d19f9jev] [] Decoding ctx of RPC function 'kfp.upload_pipeline'
2021-03-01 11:55:38 run:95 [[DEBUG]] [TID=q0d19f9jev] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'kfp.upload_pipeline'
2021-03-01 11:55:38 run:104 [[DEBUG]] [TID=q0d19f9jev] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'kfp.upload_pipeline'
2021-03-01 11:55:38 run:114 [[INFO]] [TID=q0d19f9jev] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'upload_pipeline(pipeline_package_path=newpipeline-tko28.pipeline.yaml, pipeline_metadata={'experiment_name': 'calculation exppriments', 'pipeline_name': 'newpipeline-tko28', 'pipeline_description': 'this is test pipeline', 'pipeline_args': '', 'pipeline_args_names': '', 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'volumes': [], 'abs_working_dir': '/home/jovyan', 'marshal_volume': True, 'marshal_path': '/marshal', 'snapshot_volumes': False, 'autosnapshot': False, 'steps_defaults': [], 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'volume_access_mode': 'rwm', 'parameters_names': [], 'parameters_types': [], 'parameters_values': []}, overwrite=False)'
2021-03-01 11:55:38 run:125 [[ERROR]] [TID=q0d19f9jev] [/home/jovyan/Simple Notebook Pipeline.ipynb] RPC function 'upload_pipeline' raised an unhandled exception
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run
result = func(request, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 93, in upload_pipeline
client = _get_client()
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 27, in _get_client
_client = kfp.Client()
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 119, in init
self._configure_incluster_auth()
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 727, in _configure_incluster_auth
self.set_user_namespace(old_ns_setting or _get_pod_namespace())
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 239, in set_user_namespace
with open(Client.LOCAL_KFP_CONTEXT, 'w') as f:
PermissionError: [Errno 13] Permission denied: '/home/jovyan/.config/kfp/context.json'
2021-03-01 12:00:01 run:83 [[DEBUG]] [TID=rvov88tmid] [] Decoding ctx of RPC function 'nb.explore_notebook'
2021-03-01 12:00:01 run:95 [[DEBUG]] [TID=rvov88tmid] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.explore_notebook'
2021-03-01 12:00:01 run:104 [[DEBUG]] [TID=rvov88tmid] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.explore_notebook'
2021-03-01 12:00:01 run:114 [[INFO]] [TID=rvov88tmid] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'explore_notebook(source_notebook_path=/home/jovyan/upstream-lightweight.ipynb)'
2021-03-01 12:00:02 run:83 [[DEBUG]] [TID=vyqjly0bvm] [] Decoding ctx of RPC function 'nb.get_base_image'
2021-03-01 12:00:02 run:95 [[DEBUG]] [TID=vyqjly0bvm] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.get_base_image'
2021-03-01 12:00:02 run:104 [[DEBUG]] [TID=vyqjly0bvm] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.get_base_image'
2021-03-01 12:00:02 run:114 [[INFO]] [TID=vyqjly0bvm] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'get_base_image()'
2021-03-01 12:00:02 run:83 [[DEBUG]] [TID=1l989i2kh5] [] Decoding ctx of RPC function 'nb.find_poddefault_labels_on_server'
2021-03-01 12:00:02 run:95 [[DEBUG]] [TID=1l989i2kh5] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.find_poddefault_labels_on_server'
2021-03-01 12:00:02 run:104 [[DEBUG]] [TID=1l989i2kh5] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.find_poddefault_labels_on_server'
2021-03-01 12:00:02 run:114 [[INFO]] [TID=1l989i2kh5] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'find_poddefault_labels_on_server()'
2021-03-01 12:00:02 nb:205 [[INFO]] [TID=1l989i2kh5] [/home/jovyan/upstream-lightweight.ipynb] Retrieving PodDefaults applied to server...
2021-03-01 12:00:02 nb:211 [[INFO]] [TID=1l989i2kh5] [/home/jovyan/upstream-lightweight.ipynb] Retrieved applied PodDefaults: []
2021-03-01 12:00:02 nb:215 [[INFO]] [TID=1l989i2kh5] [/home/jovyan/upstream-lightweight.ipynb] PodDefault labels applied on server:
2021-03-01 12:00:02 run:83 [[DEBUG]] [TID=22s6a7ugxe] [] Decoding ctx of RPC function 'kfp.list_experiments'
2021-03-01 12:00:02 run:95 [[DEBUG]] [TID=22s6a7ugxe] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'kfp.list_experiments'
2021-03-01 12:00:02 run:104 [[DEBUG]] [TID=22s6a7ugxe] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'kfp.list_experiments'
2021-03-01 12:00:02 run:114 [[INFO]] [TID=22s6a7ugxe] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'list_experiments()'
2021-03-01 12:00:02 run:125 [[ERROR]] [TID=22s6a7ugxe] [/home/jovyan/upstream-lightweight.ipynb] RPC function 'list_experiments' raised an unhandled exception
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run
result = func(request, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 34, in list_experiments
c = _get_client()
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 27, in _get_client
_client = kfp.Client()
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 119, in init
self._configure_incluster_auth()
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 727, in _configure_incluster_auth
self.set_user_namespace(old_ns_setting or _get_pod_namespace())
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 239, in set_user_namespace
with open(Client.LOCAL_KFP_CONTEXT, 'w') as f:
PermissionError: [Errno 13] Permission denied: '/home/jovyan/.config/kfp/context.json'
2021-03-01 12:00:24 run:83 [[DEBUG]] [TID=mj1gglvxxj] [] Decoding ctx of RPC function 'nb.explore_notebook'
2021-03-01 12:00:24 run:95 [[DEBUG]] [TID=mj1gglvxxj] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.explore_notebook'
2021-03-01 12:00:24 run:104 [[DEBUG]] [TID=mj1gglvxxj] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.explore_notebook'
2021-03-01 12:00:24 run:114 [[INFO]] [TID=mj1gglvxxj] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'explore_notebook(source_notebook_path=/home/jovyan/upstream-lightweight.ipynb)'
2021-03-01 12:00:24 run:83 [[DEBUG]] [TID=nd9wykhgs9] [] Decoding ctx of RPC function 'nb.get_base_image'
2021-03-01 12:00:24 run:95 [[DEBUG]] [TID=nd9wykhgs9] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.get_base_image'
2021-03-01 12:00:24 run:104 [[DEBUG]] [TID=nd9wykhgs9] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.get_base_image'
2021-03-01 12:00:24 run:114 [[INFO]] [TID=nd9wykhgs9] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'get_base_image()'
2021-03-01 12:00:24 run:83 [[DEBUG]] [TID=kwsd5bf0jm] [] Decoding ctx of RPC function 'nb.find_poddefault_labels_on_server'
2021-03-01 12:00:24 run:95 [[DEBUG]] [TID=kwsd5bf0jm] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.find_poddefault_labels_on_server'
2021-03-01 12:00:24 run:104 [[DEBUG]] [TID=kwsd5bf0jm] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.find_poddefault_labels_on_server'
2021-03-01 12:00:24 run:114 [[INFO]] [TID=kwsd5bf0jm] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'find_poddefault_labels_on_server()'
2021-03-01 12:00:24 nb:205 [[INFO]] [TID=kwsd5bf0jm] [/home/jovyan/upstream-lightweight.ipynb] Retrieving PodDefaults applied to server...
2021-03-01 12:00:24 nb:211 [[INFO]] [TID=kwsd5bf0jm] [/home/jovyan/upstream-lightweight.ipynb] Retrieved applied PodDefaults: []
2021-03-01 12:00:24 nb:215 [[INFO]] [TID=kwsd5bf0jm] [/home/jovyan/upstream-lightweight.ipynb] PodDefault labels applied on server:
2021-03-01 12:00:25 run:83 [[DEBUG]] [TID=xkqxkjz8u0] [] Decoding ctx of RPC function 'kfp.list_experiments'
2021-03-01 12:00:25 run:95 [[DEBUG]] [TID=xkqxkjz8u0] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'kfp.list_experiments'
2021-03-01 12:00:25 run:104 [[DEBUG]] [TID=xkqxkjz8u0] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'kfp.list_experiments'
2021-03-01 12:00:25 run:114 [[INFO]] [TID=xkqxkjz8u0] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'list_experiments()'
2021-03-01 12:00:25 run:125 [[ERROR]] [TID=xkqxkjz8u0] [/home/jovyan/upstream-lightweight.ipynb] RPC function 'list_experiments' raised an unhandled exception
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run
result = func(request, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 34, in list_experiments
c = _get_client()
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 27, in _get_client
_client = kfp.Client()
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 119, in init
self._configure_incluster_auth()
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 727, in _configure_incluster_auth
self.set_user_namespace(old_ns_setting or _get_pod_namespace())
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 239, in set_user_namespace
with open(Client.LOCAL_KFP_CONTEXT, 'w') as f:
PermissionError: [Errno 13] Permission denied: '/home/jovyan/.config/kfp/context.json'
2021-03-01 12:00:47 run:83 [[DEBUG]] [TID=z1xgerubhz] [] Decoding ctx of RPC function 'nb.validate_notebook'
2021-03-01 12:00:47 run:95 [[DEBUG]] [TID=z1xgerubhz] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook'
2021-03-01 12:00:47 run:104 [[DEBUG]] [TID=z1xgerubhz] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.validate_notebook'
2021-03-01 12:00:47 run:114 [[INFO]] [TID=z1xgerubhz] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/upstream-lightweight.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'name': 'new-ep', 'id': 'new'}, 'experiment_name': 'new-ep', 'katib_run': False, 'pipeline_description': '', 'pipeline_name': '', 'snapshot_volumes': False, 'steps_defaults': [], 'volumes': [], 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}})'
2021-03-01 12:00:47 run:125 [[ERROR]] [TID=z1xgerubhz] [/home/jovyan/upstream-lightweight.ipynb] RPC function 'validate_notebook' raised an unhandled exception
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run
result = func(request, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/nb.py", line 107, in validate_notebook
Kale(source_notebook_path, notebook_metadata_overrides)
File "/usr/local/lib/python3.6/dist-packages/kale/core.py", line 61, in init
self.pipeline_metadata = parse_metadata(notebook_metadata)
File "/usr/local/lib/python3.6/dist-packages/kale/common/metadatautils.py", line 74, in parse_metadata
raise ValueError("Pipeline name {}".format(KALE_NAME_MSG))
ValueError: Pipeline name must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character.
2021-03-01 12:01:05 run:83 [[DEBUG]] [TID=gh1j4snbz8] [] Decoding ctx of RPC function 'nb.validate_notebook'
2021-03-01 12:01:05 run:95 [[DEBUG]] [TID=gh1j4snbz8] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook'
2021-03-01 12:01:05 run:104 [[DEBUG]] [TID=gh1j4snbz8] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.validate_notebook'
2021-03-01 12:01:05 run:114 [[INFO]] [TID=gh1j4snbz8] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/upstream-lightweight.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'name': 'asdfasdf', 'id': 'new'}, 'experiment_name': 'asdfasdf', 'katib_run': False, 'pipeline_description': '', 'pipeline_name': 'sdfasdf', 'snapshot_volumes': False, 'steps_defaults': [], 'volumes': [], 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}})'
2021-03-01 12:01:05 run:83 [[DEBUG]] [TID=b5yfbhtrdt] [] Decoding ctx of RPC function 'nb.compile_notebook'
2021-03-01 12:01:05 run:95 [[DEBUG]] [TID=b5yfbhtrdt] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.compile_notebook'
2021-03-01 12:01:05 run:104 [[DEBUG]] [TID=b5yfbhtrdt] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.compile_notebook'
2021-03-01 12:01:05 run:114 [[INFO]] [TID=b5yfbhtrdt] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'compile_notebook(source_notebook_path=/home/jovyan/upstream-lightweight.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'name': 'asdfasdf', 'id': 'new'}, 'experiment_name': 'asdfasdf', 'katib_run': False, 'pipeline_description': '', 'pipeline_name': 'sdfasdf', 'snapshot_volumes': False, 'steps_defaults': [], 'volumes': [], 'katib_metadata': {'parameters': [], 'objective': {'type': 'minimize', 'objectiveMetricName': ''}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}}, debug=False)'
2021-03-01 12:01:05 run:125 [[ERROR]] [TID=b5yfbhtrdt] [/home/jovyan/upstream-lightweight.ipynb] RPC function 'compile_notebook' raised an unhandled exception
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run
result = func(request, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/nb.py", line 92, in compile_notebook
pipeline_graph, pipeline_parameters = instance.notebook_to_graph()
File "/usr/local/lib/python3.6/dist-packages/kale/core.py", line 102, in notebook_to_graph
self.pipeline_metadata)
File "/usr/local/lib/python3.6/dist-packages/kale/nbparser/parser.py", line 305, in parse_notebook
tags = parse_metadata(c.metadata)
File "/usr/local/lib/python3.6/dist-packages/kale/nbparser/parser.py", line 95, in parse_metadata
raise ValueError("Unrecognized tag: {}".format(t))
ValueError: Unrecognized tag: parameters
2021-03-01 12:01:53 run:83 [[DEBUG]] [TID=o2nglb8cvr] [] Decoding ctx of RPC function 'nb.explore_notebook'
2021-03-01 12:01:53 run:95 [[DEBUG]] [TID=o2nglb8cvr] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.explore_notebook'
2021-03-01 12:01:53 run:104 [[DEBUG]] [TID=o2nglb8cvr] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.explore_notebook'
2021-03-01 12:01:53 run:114 [[INFO]] [TID=o2nglb8cvr] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'explore_notebook(source_notebook_path=/home/jovyan/upstream-lightweight.ipynb)'
2021-03-01 12:01:53 run:83 [[DEBUG]] [TID=kqml0x4kf1] [] Decoding ctx of RPC function 'nb.get_base_image'
2021-03-01 12:01:53 run:95 [[DEBUG]] [TID=kqml0x4kf1] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.get_base_image'
2021-03-01 12:01:53 run:104 [[DEBUG]] [TID=kqml0x4kf1] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.get_base_image'
2021-03-01 12:01:53 run:114 [[INFO]] [TID=kqml0x4kf1] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'get_base_image()'
2021-03-01 12:01:53 run:83 [[DEBUG]] [TID=von88m6u2t] [] Decoding ctx of RPC function 'nb.find_poddefault_labels_on_server'
2021-03-01 12:01:53 run:95 [[DEBUG]] [TID=von88m6u2t] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.find_poddefault_labels_on_server'
2021-03-01 12:01:53 run:104 [[DEBUG]] [TID=von88m6u2t] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.find_poddefault_labels_on_server'
2021-03-01 12:01:53 run:114 [[INFO]] [TID=von88m6u2t] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'find_poddefault_labels_on_server()'
2021-03-01 12:01:53 nb:205 [[INFO]] [TID=von88m6u2t] [/home/jovyan/upstream-lightweight.ipynb] Retrieving PodDefaults applied to server...
2021-03-01 12:01:54 nb:211 [[INFO]] [TID=von88m6u2t] [/home/jovyan/upstream-lightweight.ipynb] Retrieved applied PodDefaults: []
2021-03-01 12:01:54 nb:215 [[INFO]] [TID=von88m6u2t] [/home/jovyan/upstream-lightweight.ipynb] PodDefault labels applied on server:
2021-03-01 12:01:54 run:83 [[DEBUG]] [TID=bt1jayopsc] [] Decoding ctx of RPC function 'kfp.list_experiments'
2021-03-01 12:01:54 run:95 [[DEBUG]] [TID=bt1jayopsc] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'kfp.list_experiments'
2021-03-01 12:01:54 run:104 [[DEBUG]] [TID=bt1jayopsc] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'kfp.list_experiments'
2021-03-01 12:01:54 run:114 [[INFO]] [TID=bt1jayopsc] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'list_experiments()'
2021-03-01 12:01:54 run:125 [[ERROR]] [TID=bt1jayopsc] [/home/jovyan/upstream-lightweight.ipynb] RPC function 'list_experiments' raised an unhandled exception
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run
result = func(request, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 34, in list_experiments
c = _get_client()
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 27, in _get_client
_client = kfp.Client()
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 119, in init
self._configure_incluster_auth()
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 727, in _configure_incluster_auth
self.set_user_namespace(old_ns_setting or _get_pod_namespace())
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 239, in set_user_namespace
with open(Client.LOCAL_KFP_CONTEXT, 'w') as f:
PermissionError: [Errno 13] Permission denied: '/home/jovyan/.config/kfp/context.json'
2021-03-01 12:04:47 run:83 [[DEBUG]] [TID=a6sfukms9q] [] Decoding ctx of RPC function 'nb.explore_notebook'
2021-03-01 12:04:47 run:95 [[DEBUG]] [TID=a6sfukms9q] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.explore_notebook'
2021-03-01 12:04:47 run:104 [[DEBUG]] [TID=a6sfukms9q] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.explore_notebook'
2021-03-01 12:04:47 run:114 [[INFO]] [TID=a6sfukms9q] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'explore_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb)'
2021-03-01 12:04:47 run:83 [[DEBUG]] [TID=03c3vvp33c] [] Decoding ctx of RPC function 'nb.get_base_image'
2021-03-01 12:04:47 run:95 [[DEBUG]] [TID=03c3vvp33c] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.get_base_image'
2021-03-01 12:04:47 run:104 [[DEBUG]] [TID=03c3vvp33c] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.get_base_image'
2021-03-01 12:04:47 run:114 [[INFO]] [TID=03c3vvp33c] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'get_base_image()'
2021-03-01 12:04:48 run:83 [[DEBUG]] [TID=sna33bbvqp] [] Decoding ctx of RPC function 'nb.find_poddefault_labels_on_server'
2021-03-01 12:04:48 run:95 [[DEBUG]] [TID=sna33bbvqp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.find_poddefault_labels_on_server'
2021-03-01 12:04:48 run:104 [[DEBUG]] [TID=sna33bbvqp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.find_poddefault_labels_on_server'
2021-03-01 12:04:48 run:114 [[INFO]] [TID=sna33bbvqp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'find_poddefault_labels_on_server()'
2021-03-01 12:04:48 nb:205 [[INFO]] [TID=sna33bbvqp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Retrieving PodDefaults applied to server...
2021-03-01 12:04:48 nb:211 [[INFO]] [TID=sna33bbvqp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Retrieved applied PodDefaults: []
2021-03-01 12:04:48 nb:215 [[INFO]] [TID=sna33bbvqp] [/home/jovyan/Simple Notebook Pipeline.ipynb] PodDefault labels applied on server:
2021-03-01 12:04:48 run:83 [[DEBUG]] [TID=abv7oeldfk] [] Decoding ctx of RPC function 'kfp.list_experiments'
2021-03-01 12:04:48 run:95 [[DEBUG]] [TID=abv7oeldfk] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'kfp.list_experiments'
2021-03-01 12:04:48 run:104 [[DEBUG]] [TID=abv7oeldfk] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'kfp.list_experiments'
2021-03-01 12:04:48 run:114 [[INFO]] [TID=abv7oeldfk] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'list_experiments()'
2021-03-01 12:04:53 run:83 [[DEBUG]] [TID=7e8qrens2k] [] Decoding ctx of RPC function 'nb.validate_notebook'
2021-03-01 12:04:53 run:95 [[DEBUG]] [TID=7e8qrens2k] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook'
2021-03-01 12:04:53 run:104 [[DEBUG]] [TID=7e8qrens2k] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.validate_notebook'
2021-03-01 12:04:53 run:114 [[INFO]] [TID=7e8qrens2k] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'experiment_name': 'calculation exppriments', 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'pipeline_description': 'this is test pipeline', 'pipeline_name': 'newpipeline', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': []})'
2021-03-01 12:04:54 run:83 [[DEBUG]] [TID=1lagbopnng] [] Decoding ctx of RPC function 'nb.compile_notebook'
2021-03-01 12:04:54 run:95 [[DEBUG]] [TID=1lagbopnng] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.compile_notebook'
2021-03-01 12:04:54 run:104 [[DEBUG]] [TID=1lagbopnng] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.compile_notebook'
2021-03-01 12:04:54 run:114 [[INFO]] [TID=1lagbopnng] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'compile_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'experiment_name': 'calculation exppriments', 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'pipeline_description': 'this is test pipeline', 'pipeline_name': 'newpipeline', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': []}, debug=False)'
2021-03-01 12:04:54 core:146 [[DEBUG]] [TID=1lagbopnng] [/home/jovyan/Simple Notebook Pipeline.ipynb] ------------- Kale Start Run -------------
2021-03-01 12:04:54 core:205 [[INFO]] [TID=1lagbopnng] [/home/jovyan/Simple Notebook Pipeline.ipynb] Pipeline code saved at /home/jovyan/newpipeline-yl6ca.kale.py
2021-03-01 12:04:54 run:83 [[DEBUG]] [TID=ieogqedcee] [] Decoding ctx of RPC function 'kfp.upload_pipeline'
2021-03-01 12:04:54 run:95 [[DEBUG]] [TID=ieogqedcee] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'kfp.upload_pipeline'
2021-03-01 12:04:54 run:104 [[DEBUG]] [TID=ieogqedcee] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'kfp.upload_pipeline'
2021-03-01 12:04:54 run:114 [[INFO]] [TID=ieogqedcee] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'upload_pipeline(pipeline_package_path=newpipeline-yl6ca.pipeline.yaml, pipeline_metadata={'experiment_name': 'calculation exppriments', 'pipeline_name': 'newpipeline-yl6ca', 'pipeline_description': 'this is test pipeline', 'pipeline_args': '', 'pipeline_args_names': '', 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'volumes': [], 'abs_working_dir': '/home/jovyan', 'marshal_volume': True, 'marshal_path': '/marshal', 'snapshot_volumes': False, 'autosnapshot': False, 'steps_defaults': [], 'experiment': {'id': '366339c7-0970-461c-a8b8-b2c14b96f6b8', 'name': 'calculation exppriments'}, 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'volume_access_mode': 'rwm', 'parameters_names': [], 'parameters_types': [], 'parameters_values': []}, overwrite=False)'
2021-03-01 12:04:55 run:125 [[ERROR]] [TID=ieogqedcee] [/home/jovyan/Simple Notebook Pipeline.ipynb] RPC function 'upload_pipeline' raised an unhandled exception
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run
result = func(request, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 96, in upload_pipeline
pipeline = client.upload_pipeline(pipeline_package_path, pipeline_name)
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 692, in upload_pipeline
response = self._upload_api.upload_pipeline(pipeline_package_path, name=pipeline_name, description=description)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api/pipeline_upload_service_api.py", line 56, in upload_pipeline
(data) = self.upload_pipeline_with_http_info(uploadfile, **kwargs) # noqa: E501
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api/pipeline_upload_service_api.py", line 139, in upload_pipeline_with_http_info
collection_formats=collection_formats)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 330, in call_api
_preload_content, _request_timeout)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 169, in __call_api
return_data = self.deserialize(response_data, response_type)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 241, in deserialize
return self.__deserialize(data, response_type)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 280, in __deserialize
return self.__deserialize_model(data, klass)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 624, in __deserialize_model
kwargs[attr] = self.__deserialize(value, attr_type)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 280, in __deserialize
return self.__deserialize_model(data, klass)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 624, in __deserialize_model
kwargs[attr] = self.__deserialize(value, attr_type)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 278, in __deserialize
return self.__deserialize_datatime(data)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 590, in __deserialize_datatime
return parse(string)
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 1374, in parse
return DEFAULTPARSER.parse(timestr, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 646, in parse
res, skipped_tokens = self._parse(timestr, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 725, in _parse
l = _timelex.split(timestr) # Splits the timestr into tokens
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 207, in split
return list(cls(s))
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 76, in init
'{itype}'.format(itype=instream.class.name))
TypeError: Parser must be a string or character stream, not dict
2021-03-01 12:05:16 run:83 [[DEBUG]] [TID=61slb68pwk] [] Decoding ctx of RPC function 'nb.validate_notebook'
2021-03-01 12:05:16 run:95 [[DEBUG]] [TID=61slb68pwk] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook'
2021-03-01 12:05:16 run:104 [[DEBUG]] [TID=61slb68pwk] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.validate_notebook'
2021-03-01 12:05:16 run:114 [[INFO]] [TID=61slb68pwk] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'name': 'aaa', 'id': 'new'}, 'experiment_name': 'aaa', 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'pipeline_description': 'kjkjjkj', 'pipeline_name': 'newpipeline1', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': []})'
2021-03-01 12:05:17 run:83 [[DEBUG]] [TID=7bcpkmjb2t] [] Decoding ctx of RPC function 'nb.compile_notebook'
2021-03-01 12:05:17 run:95 [[DEBUG]] [TID=7bcpkmjb2t] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.compile_notebook'
2021-03-01 12:05:17 run:104 [[DEBUG]] [TID=7bcpkmjb2t] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.compile_notebook'
2021-03-01 12:05:17 run:114 [[INFO]] [TID=7bcpkmjb2t] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'compile_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'name': 'aaa', 'id': 'new'}, 'experiment_name': 'aaa', 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'pipeline_description': 'kjkjjkj', 'pipeline_name': 'newpipeline1', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': []}, debug=False)'
2021-03-01 12:05:17 core:146 [[DEBUG]] [TID=7bcpkmjb2t] [/home/jovyan/Simple Notebook Pipeline.ipynb] ------------- Kale Start Run -------------
2021-03-01 12:05:17 core:205 [[INFO]] [TID=7bcpkmjb2t] [/home/jovyan/Simple Notebook Pipeline.ipynb] Pipeline code saved at /home/jovyan/newpipeline1-o3pfg.kale.py
2021-03-01 12:05:17 run:83 [[DEBUG]] [TID=sjcjhls9p4] [] Decoding ctx of RPC function 'kfp.upload_pipeline'
2021-03-01 12:05:17 run:95 [[DEBUG]] [TID=sjcjhls9p4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'kfp.upload_pipeline'
2021-03-01 12:05:17 run:104 [[DEBUG]] [TID=sjcjhls9p4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'kfp.upload_pipeline'
2021-03-01 12:05:17 run:114 [[INFO]] [TID=sjcjhls9p4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'upload_pipeline(pipeline_package_path=newpipeline1-o3pfg.pipeline.yaml, pipeline_metadata={'experiment_name': 'aaa', 'pipeline_name': 'newpipeline1-o3pfg', 'pipeline_description': 'kjkjjkj', 'pipeline_args': '', 'pipeline_args_names': '', 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'volumes': [], 'abs_working_dir': '/home/jovyan', 'marshal_volume': True, 'marshal_path': '/marshal', 'snapshot_volumes': False, 'autosnapshot': False, 'steps_defaults': [], 'experiment': {'name': 'aaa', 'id': 'new'}, 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'volume_access_mode': 'rwm', 'parameters_names': [], 'parameters_types': [], 'parameters_values': []}, overwrite=False)'
2021-03-01 12:05:18 run:125 [[ERROR]] [TID=sjcjhls9p4] [/home/jovyan/Simple Notebook Pipeline.ipynb] RPC function 'upload_pipeline' raised an unhandled exception
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run
result = func(request, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 96, in upload_pipeline
pipeline = client.upload_pipeline(pipeline_package_path, pipeline_name)
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 692, in upload_pipeline
response = self._upload_api.upload_pipeline(pipeline_package_path, name=pipeline_name, description=description)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api/pipeline_upload_service_api.py", line 56, in upload_pipeline
(data) = self.upload_pipeline_with_http_info(uploadfile, **kwargs) # noqa: E501
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api/pipeline_upload_service_api.py", line 139, in upload_pipeline_with_http_info
collection_formats=collection_formats)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 330, in call_api
_preload_content, _request_timeout)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 169, in __call_api
return_data = self.deserialize(response_data, response_type)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 241, in deserialize
return self.__deserialize(data, response_type)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 280, in __deserialize
return self.__deserialize_model(data, klass)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 624, in __deserialize_model
kwargs[attr] = self.__deserialize(value, attr_type)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 280, in __deserialize
return self.__deserialize_model(data, klass)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 624, in __deserialize_model
kwargs[attr] = self.__deserialize(value, attr_type)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 278, in __deserialize
return self.__deserialize_datatime(data)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 590, in __deserialize_datatime
return parse(string)
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 1374, in parse
return DEFAULTPARSER.parse(timestr, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 646, in parse
res, skipped_tokens = self._parse(timestr, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 725, in _parse
l = _timelex.split(timestr) # Splits the timestr into tokens
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 207, in split
return list(cls(s))
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 76, in init
'{itype}'.format(itype=instream.class.name))
TypeError: Parser must be a string or character stream, not dict
2021-03-01 12:06:26 run:83 [[DEBUG]] [TID=cpwxpbicsc] [] Decoding ctx of RPC function 'nb.validate_notebook'
2021-03-01 12:06:26 run:95 [[DEBUG]] [TID=cpwxpbicsc] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook'
2021-03-01 12:06:26 run:104 [[DEBUG]] [TID=cpwxpbicsc] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.validate_notebook'
2021-03-01 12:06:26 run:114 [[INFO]] [TID=cpwxpbicsc] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'name': 'aaa', 'id': 'new'}, 'experiment_name': 'aaa', 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'pipeline_description': 'kjkjjkj', 'pipeline_name': 'newpipeline1', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': []})'
2021-03-01 12:06:26 run:83 [[DEBUG]] [TID=96geeq1vxj] [] Decoding ctx of RPC function 'nb.compile_notebook'
2021-03-01 12:06:26 run:95 [[DEBUG]] [TID=96geeq1vxj] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.compile_notebook'
2021-03-01 12:06:26 run:104 [[DEBUG]] [TID=96geeq1vxj] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.compile_notebook'
2021-03-01 12:06:26 run:114 [[INFO]] [TID=96geeq1vxj] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'compile_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'name': 'aaa', 'id': 'new'}, 'experiment_name': 'aaa', 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'pipeline_description': 'kjkjjkj', 'pipeline_name': 'newpipeline1', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': []}, debug=False)'
2021-03-01 12:06:26 core:146 [[DEBUG]] [TID=96geeq1vxj] [/home/jovyan/Simple Notebook Pipeline.ipynb] ------------- Kale Start Run -------------
2021-03-01 12:06:27 core:205 [[INFO]] [TID=96geeq1vxj] [/home/jovyan/Simple Notebook Pipeline.ipynb] Pipeline code saved at /home/jovyan/newpipeline1-e0ggq.kale.py
2021-03-01 12:06:27 run:83 [[DEBUG]] [TID=aazcq2llyl] [] Decoding ctx of RPC function 'kfp.upload_pipeline'
2021-03-01 12:06:27 run:95 [[DEBUG]] [TID=aazcq2llyl] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'kfp.upload_pipeline'
2021-03-01 12:06:27 run:104 [[DEBUG]] [TID=aazcq2llyl] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'kfp.upload_pipeline'
2021-03-01 12:06:27 run:114 [[INFO]] [TID=aazcq2llyl] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'upload_pipeline(pipeline_package_path=newpipeline1-e0ggq.pipeline.yaml, pipeline_metadata={'experiment_name': 'aaa', 'pipeline_name': 'newpipeline1-e0ggq', 'pipeline_description': 'kjkjjkj', 'pipeline_args': '', 'pipeline_args_names': '', 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'volumes': [], 'abs_working_dir': '/home/jovyan', 'marshal_volume': True, 'marshal_path': '/marshal', 'snapshot_volumes': False, 'autosnapshot': False, 'steps_defaults': [], 'experiment': {'name': 'aaa', 'id': 'new'}, 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'volume_access_mode': 'rwm', 'parameters_names': [], 'parameters_types': [], 'parameters_values': []}, overwrite=False)'
2021-03-01 12:06:27 run:125 [[ERROR]] [TID=aazcq2llyl] [/home/jovyan/Simple Notebook Pipeline.ipynb] RPC function 'upload_pipeline' raised an unhandled exception
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run
result = func(request, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 96, in upload_pipeline
pipeline = client.upload_pipeline(pipeline_package_path, pipeline_name)
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 692, in upload_pipeline
response = self._upload_api.upload_pipeline(pipeline_package_path, name=pipeline_name, description=description)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api/pipeline_upload_service_api.py", line 56, in upload_pipeline
(data) = self.upload_pipeline_with_http_info(uploadfile, **kwargs) # noqa: E501
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api/pipeline_upload_service_api.py", line 139, in upload_pipeline_with_http_info
collection_formats=collection_formats)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 330, in call_api
_preload_content, _request_timeout)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 169, in __call_api
return_data = self.deserialize(response_data, response_type)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 241, in deserialize
return self.__deserialize(data, response_type)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 280, in __deserialize
return self.__deserialize_model(data, klass)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 624, in __deserialize_model
kwargs[attr] = self.__deserialize(value, attr_type)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 280, in __deserialize
return self.__deserialize_model(data, klass)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 624, in __deserialize_model
kwargs[attr] = self.__deserialize(value, attr_type)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 278, in __deserialize
return self.__deserialize_datatime(data)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 590, in __deserialize_datatime
return parse(string)
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 1374, in parse
return DEFAULTPARSER.parse(timestr, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 646, in parse
res, skipped_tokens = self._parse(timestr, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 725, in _parse
l = _timelex.split(timestr) # Splits the timestr into tokens
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 207, in split
return list(cls(s))
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 76, in init
'{itype}'.format(itype=instream.class.name))
TypeError: Parser must be a string or character stream, not dict
2021-03-01 12:07:22 run:83 [[DEBUG]] [TID=dujb1945gv] [] Decoding ctx of RPC function 'nb.validate_notebook'
2021-03-01 12:07:22 run:95 [[DEBUG]] [TID=dujb1945gv] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook'
2021-03-01 12:07:22 run:104 [[DEBUG]] [TID=dujb1945gv] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.validate_notebook'
2021-03-01 12:07:22 run:114 [[INFO]] [TID=dujb1945gv] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'name': 'aaa', 'id': 'new'}, 'experiment_name': 'aaa', 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'pipeline_description': 'kjkjjkj', 'pipeline_name': 'newpipeline1', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': []})'
2021-03-01 12:07:22 run:83 [[DEBUG]] [TID=g4c8akha7j] [] Decoding ctx of RPC function 'nb.compile_notebook'
2021-03-01 12:07:22 run:95 [[DEBUG]] [TID=g4c8akha7j] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.compile_notebook'
2021-03-01 12:07:22 run:104 [[DEBUG]] [TID=g4c8akha7j] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.compile_notebook'
2021-03-01 12:07:22 run:114 [[INFO]] [TID=g4c8akha7j] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'compile_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'name': 'aaa', 'id': 'new'}, 'experiment_name': 'aaa', 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'pipeline_description': 'kjkjjkj', 'pipeline_name': 'newpipeline1', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': []}, debug=False)'
2021-03-01 12:07:22 core:146 [[DEBUG]] [TID=g4c8akha7j] [/home/jovyan/Simple Notebook Pipeline.ipynb] ------------- Kale Start Run -------------
2021-03-01 12:07:23 core:205 [[INFO]] [TID=g4c8akha7j] [/home/jovyan/Simple Notebook Pipeline.ipynb] Pipeline code saved at /home/jovyan/newpipeline1-3qkiy.kale.py
2021-03-01 12:07:23 run:83 [[DEBUG]] [TID=is7qozmeux] [] Decoding ctx of RPC function 'kfp.upload_pipeline'
2021-03-01 12:07:23 run:95 [[DEBUG]] [TID=is7qozmeux] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'kfp.upload_pipeline'
2021-03-01 12:07:23 run:104 [[DEBUG]] [TID=is7qozmeux] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'kfp.upload_pipeline'
2021-03-01 12:07:23 run:114 [[INFO]] [TID=is7qozmeux] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'upload_pipeline(pipeline_package_path=newpipeline1-3qkiy.pipeline.yaml, pipeline_metadata={'experiment_name': 'aaa', 'pipeline_name': 'newpipeline1-3qkiy', 'pipeline_description': 'kjkjjkj', 'pipeline_args': '', 'pipeline_args_names': '', 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'volumes': [], 'abs_working_dir': '/home/jovyan', 'marshal_volume': True, 'marshal_path': '/marshal', 'snapshot_volumes': False, 'autosnapshot': False, 'steps_defaults': [], 'experiment': {'name': 'aaa', 'id': 'new'}, 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'volume_access_mode': 'rwm', 'parameters_names': [], 'parameters_types': [], 'parameters_values': []}, overwrite=False)'
2021-03-01 12:07:23 run:125 [[ERROR]] [TID=is7qozmeux] [/home/jovyan/Simple Notebook Pipeline.ipynb] RPC function 'upload_pipeline' raised an unhandled exception
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run
result = func(request, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 96, in upload_pipeline
pipeline = client.upload_pipeline(pipeline_package_path, pipeline_name)
File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 692, in upload_pipeline
response = self._upload_api.upload_pipeline(pipeline_package_path, name=pipeline_name, description=description)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api/pipeline_upload_service_api.py", line 56, in upload_pipeline
(data) = self.upload_pipeline_with_http_info(uploadfile, **kwargs) # noqa: E501
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api/pipeline_upload_service_api.py", line 139, in upload_pipeline_with_http_info
collection_formats=collection_formats)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 330, in call_api
_preload_content, _request_timeout)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 169, in __call_api
return_data = self.deserialize(response_data, response_type)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 241, in deserialize
return self.__deserialize(data, response_type)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 280, in __deserialize
return self.__deserialize_model(data, klass)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 624, in __deserialize_model
kwargs[attr] = self.__deserialize(value, attr_type)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 280, in __deserialize
return self.__deserialize_model(data, klass)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 624, in __deserialize_model
kwargs[attr] = self.__deserialize(value, attr_type)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 278, in __deserialize
return self.__deserialize_datatime(data)
File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 590, in __deserialize_datatime
return parse(string)
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 1374, in parse
return DEFAULTPARSER.parse(timestr, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 646, in parse
res, skipped_tokens = self._parse(timestr, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 725, in _parse
l = _timelex.split(timestr) # Splits the timestr into tokens
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 207, in split
return list(cls(s))
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 76, in init
'{itype}'.format(itype=instream.class.name))
TypeError: Parser must be a string or character stream, not dict
2021-03-01 12:08:13 run:83 [[DEBUG]] [TID=bigecwbchg] [] Decoding ctx of RPC function 'nb.validate_notebook'
2021-03-01 12:08:13 run:95 [[DEBUG]] [TID=bigecwbchg] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook'
2021-03-01 12:08:13 run:104 [[DEBUG]] [TID=bigecwbchg] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.validate_notebook'
2021-03-01 12:08:13 run:114 [[INFO]] [TID=bigecwbchg] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'name': 'asdfasdfasdfasdf', 'id': 'new'}, 'experiment_name': 'asdfasdfasdfasdf', 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'pipeline_description': 'asdfasdfasdfadsf', 'pipeline_name': 'newpipeline1', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': []})'
2021-03-01 12:08:13 run:83 [[DEBUG]] [TID=kv6vuclz2u] [] Decoding ctx of RPC function 'nb.compile_notebook'
2021-03-01 12:08:13 run:95 [[DEBUG]] [TID=kv6vuclz2u] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.compile_notebook'
2021-03-01 12:08:13 run:104 [[DEBUG]] [TID=kv6vuclz2u] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.compile_notebook'
2021-03-01 12:08:13 run:114 [[INFO]] [TID=kv6vuclz2u] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'compile_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'name': 'asdfasdfasdfasdf', 'id': 'new'}, 'experiment_name': 'asdfasdfasdfasdf', 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'pipeline_description': 'asdfasdfasdfadsf', 'pipeline_name': 'newpipeline1', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': []}, debug=False)'
2021-03-01 12:08:13 core:146 [[DEBUG]] [TID=kv6vuclz2u] [/home/jovyan/Simple Notebook Pipeline.ipynb] ------------- Kale Start Run -------------
2021-03-01 12:08:13 core:205 [[INFO]] [TID=kv6vuclz2u] [/home/jovyan/Simple Notebook Pipeline.ipynb] Pipeline code saved at /home/jovyan/newpipeline1-afrqj.kale.py
2021-03-01 12:12:40 run:83 [[DEBUG]] [TID=a8lyhkspxp] [] Decoding ctx of RPC function 'nb.explore_notebook'
2021-03-01 12:12:40 run:95 [[DEBUG]] [TID=a8lyhkspxp] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.explore_notebook'
2021-03-01 12:12:40 run:104 [[DEBUG]] [TID=a8lyhkspxp] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.explore_notebook'
2021-03-01 12:12:40 run:114 [[INFO]] [TID=a8lyhkspxp] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'explore_notebook(source_notebook_path=/home/jovyan/upstream-lightweight.ipynb)'
2021-03-01 12:12:40 run:83 [[DEBUG]] [TID=ea3v1ay03k] [] Decoding ctx of RPC function 'nb.get_base_image'
2021-03-01 12:12:40 run:95 [[DEBUG]] [TID=ea3v1ay03k] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.get_base_image'
2021-03-01 12:12:40 run:104 [[DEBUG]] [TID=ea3v1ay03k] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.get_base_image'
2021-03-01 12:12:40 run:114 [[INFO]] [TID=ea3v1ay03k] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'get_base_image()'
2021-03-01 12:12:41 run:83 [[DEBUG]] [TID=ftiq7qpllf] [] Decoding ctx of RPC function 'nb.find_poddefault_labels_on_server'
2021-03-01 12:12:41 run:95 [[DEBUG]] [TID=ftiq7qpllf] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'nb.find_poddefault_labels_on_server'
2021-03-01 12:12:41 run:104 [[DEBUG]] [TID=ftiq7qpllf] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'nb.find_poddefault_labels_on_server'
2021-03-01 12:12:41 run:114 [[INFO]] [TID=ftiq7qpllf] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'find_poddefault_labels_on_server()'
2021-03-01 12:12:41 nb:205 [[INFO]] [TID=ftiq7qpllf] [/home/jovyan/upstream-lightweight.ipynb] Retrieving PodDefaults applied to server...
2021-03-01 12:12:41 nb:211 [[INFO]] [TID=ftiq7qpllf] [/home/jovyan/upstream-lightweight.ipynb] Retrieved applied PodDefaults: []
2021-03-01 12:12:41 nb:215 [[INFO]] [TID=ftiq7qpllf] [/home/jovyan/upstream-lightweight.ipynb] PodDefault labels applied on server:
2021-03-01 12:12:41 run:83 [[DEBUG]] [TID=bf8b9ie7x8] [] Decoding ctx of RPC function 'kfp.list_experiments'
2021-03-01 12:12:41 run:95 [[DEBUG]] [TID=bf8b9ie7x8] [/home/jovyan/upstream-lightweight.ipynb] Decoding kwargs of RPC function 'kfp.list_experiments'
2021-03-01 12:12:41 run:104 [[DEBUG]] [TID=bf8b9ie7x8] [/home/jovyan/upstream-lightweight.ipynb] Importing RPC function 'kfp.list_experiments'
2021-03-01 12:12:41 run:114 [[INFO]] [TID=bf8b9ie7x8] [/home/jovyan/upstream-lightweight.ipynb] Executing RPC function 'list_experiments()'
2021-03-01 12:12:51 run:83 [[DEBUG]] [TID=8ma26jsqn4] [] Decoding ctx of RPC function 'nb.explore_notebook'
2021-03-01 12:12:51 run:95 [[DEBUG]] [TID=8ma26jsqn4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.explore_notebook'
2021-03-01 12:12:51 run:104 [[DEBUG]] [TID=8ma26jsqn4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.explore_notebook'
2021-03-01 12:12:51 run:114 [[INFO]] [TID=8ma26jsqn4] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'explore_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb)'
2021-03-01 12:13:36 run:83 [[DEBUG]] [TID=zhriwapn38] [] Decoding ctx of RPC function 'rok.check_rok_availability'
2021-03-01 12:13:36 run:95 [[DEBUG]] [TID=zhriwapn38] [] Decoding kwargs of RPC function 'rok.check_rok_availability'
2021-03-01 12:13:36 run:104 [[DEBUG]] [TID=zhriwapn38] [] Importing RPC function 'rok.check_rok_availability'
2021-03-01 12:13:38 run:114 [[INFO]] [TID=zhriwapn38] [] Executing RPC function 'check_rok_availability()'
2021-03-01 12:13:38 rok:123 [[ERROR]] [TID=zhriwapn38] [] Failed to initialize RokClient
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 117, in check_rok_availability
rok = _get_client()
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 42, in _get_client
_client = RokClient()
File "build/bdist.linux-x86_64/egg/rok_gw_client/client.py", line 175, in init
ValueError: A Rok API URL must be provided either via the url' argument or the ROK_GW_URL' environment variable.
2021-03-01 12:13:38 run:120 [[ERROR]] [TID=zhriwapn38] [] RPC function 'check_rok_availability' raised an RPCError
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 117, in check_rok_availability
rok = _get_client()
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 42, in _get_client
_client = RokClient()
File "build/bdist.linux-x86_64/egg/rok_gw_client/client.py", line 175, in init
ValueError: A Rok API URL must be provided either via the url' argument or the ROK_GW_URL' environment variable.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run
result = func(request, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 126, in check_rok_availability
trans_id=request.trans_id)
kale.rpc.errors.RPCServiceUnavailableError
2021-03-01 12:13:39 run:83 [[DEBUG]] [TID=2zzlbq8ork] [] Decoding ctx of RPC function 'nb.resume_notebook_path'
2021-03-01 12:13:39 run:95 [[DEBUG]] [TID=2zzlbq8ork] [] Decoding kwargs of RPC function 'nb.resume_notebook_path'
2021-03-01 12:13:39 run:104 [[DEBUG]] [TID=2zzlbq8ork] [] Importing RPC function 'nb.resume_notebook_path'
2021-03-01 12:13:40 run:114 [[INFO]] [TID=2zzlbq8ork] [] Executing RPC function 'resume_notebook_path(server_root=/home/jovyan)'
2021-03-01 12:13:41 run:83 [[DEBUG]] [TID=468beo3ggg] [] Decoding ctx of RPC function 'nb.explore_notebook'
2021-03-01 12:13:41 run:95 [[DEBUG]] [TID=468beo3ggg] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.explore_notebook'
2021-03-01 12:13:41 run:104 [[DEBUG]] [TID=468beo3ggg] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.explore_notebook'
2021-03-01 12:13:41 run:114 [[INFO]] [TID=468beo3ggg] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'explore_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb)'
2021-03-01 12:13:41 run:83 [[DEBUG]] [TID=18t185jz2q] [] Decoding ctx of RPC function 'nb.get_base_image'
2021-03-01 12:13:41 run:95 [[DEBUG]] [TID=18t185jz2q] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.get_base_image'
2021-03-01 12:13:41 run:104 [[DEBUG]] [TID=18t185jz2q] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.get_base_image'
2021-03-01 12:13:41 run:114 [[INFO]] [TID=18t185jz2q] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'get_base_image()'
2021-03-01 12:13:41 run:83 [[DEBUG]] [TID=zbymr0zgsp] [] Decoding ctx of RPC function 'nb.find_poddefault_labels_on_server'
2021-03-01 12:13:41 run:95 [[DEBUG]] [TID=zbymr0zgsp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.find_poddefault_labels_on_server'
2021-03-01 12:13:41 run:104 [[DEBUG]] [TID=zbymr0zgsp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.find_poddefault_labels_on_server'
2021-03-01 12:13:41 run:114 [[INFO]] [TID=zbymr0zgsp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'find_poddefault_labels_on_server()'
2021-03-01 12:13:41 nb:205 [[INFO]] [TID=zbymr0zgsp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Retrieving PodDefaults applied to server...
2021-03-01 12:13:41 nb:211 [[INFO]] [TID=zbymr0zgsp] [/home/jovyan/Simple Notebook Pipeline.ipynb] Retrieved applied PodDefaults: []
2021-03-01 12:13:41 nb:215 [[INFO]] [TID=zbymr0zgsp] [/home/jovyan/Simple Notebook Pipeline.ipynb] PodDefault labels applied on server:
2021-03-01 12:13:41 run:83 [[DEBUG]] [TID=mlxm0ps9wi] [] Decoding ctx of RPC function 'kfp.list_experiments'
2021-03-01 12:13:41 run:95 [[DEBUG]] [TID=mlxm0ps9wi] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'kfp.list_experiments'
2021-03-01 12:13:41 run:104 [[DEBUG]] [TID=mlxm0ps9wi] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'kfp.list_experiments'
2021-03-01 12:13:41 run:114 [[INFO]] [TID=mlxm0ps9wi] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'list_experiments()'
2021-03-01 12:31:12 run:83 [[DEBUG]] [TID=byphdyfl4h] [] Decoding ctx of RPC function 'rok.check_rok_availability'
2021-03-01 12:31:12 run:95 [[DEBUG]] [TID=byphdyfl4h] [] Decoding kwargs of RPC function 'rok.check_rok_availability'
2021-03-01 12:31:12 run:104 [[DEBUG]] [TID=byphdyfl4h] [] Importing RPC function 'rok.check_rok_availability'
2021-03-01 12:31:14 run:114 [[INFO]] [TID=byphdyfl4h] [] Executing RPC function 'check_rok_availability()'
2021-03-01 12:31:14 rok:123 [[ERROR]] [TID=byphdyfl4h] [] Failed to initialize RokClient
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 117, in check_rok_availability
rok = _get_client()
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 42, in _get_client
_client = RokClient()
File "build/bdist.linux-x86_64/egg/rok_gw_client/client.py", line 175, in init
ValueError: A Rok API URL must be provided either via the url' argument or the ROK_GW_URL' environment variable.
2021-03-01 12:31:14 run:120 [[ERROR]] [TID=byphdyfl4h] [] RPC function 'check_rok_availability' raised an RPCError
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 117, in check_rok_availability
rok = _get_client()
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 42, in _get_client
_client = RokClient()
File "build/bdist.linux-x86_64/egg/rok_gw_client/client.py", line 175, in init
ValueError: A Rok API URL must be provided either via the url' argument or the ROK_GW_URL' environment variable.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run
result = func(request, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 126, in check_rok_availability
trans_id=request.trans_id)
kale.rpc.errors.RPCServiceUnavailableError
2021-03-01 12:31:15 run:83 [[DEBUG]] [TID=y9i32q6ip1] [] Decoding ctx of RPC function 'nb.resume_notebook_path'
2021-03-01 12:31:15 run:95 [[DEBUG]] [TID=y9i32q6ip1] [] Decoding kwargs of RPC function 'nb.resume_notebook_path'
2021-03-01 12:31:15 run:104 [[DEBUG]] [TID=y9i32q6ip1] [] Importing RPC function 'nb.resume_notebook_path'
2021-03-01 12:31:16 run:114 [[INFO]] [TID=y9i32q6ip1] [] Executing RPC function 'resume_notebook_path(server_root=/home/jovyan)'
2021-03-01 12:31:16 run:83 [[DEBUG]] [TID=iwbk0qequh] [] Decoding ctx of RPC function 'nb.explore_notebook'
2021-03-01 12:31:16 run:95 [[DEBUG]] [TID=iwbk0qequh] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.explore_notebook'
2021-03-01 12:31:16 run:104 [[DEBUG]] [TID=iwbk0qequh] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.explore_notebook'
2021-03-01 12:31:16 run:114 [[INFO]] [TID=iwbk0qequh] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'explore_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb)'
2021-03-01 12:31:17 run:83 [[DEBUG]] [TID=1rfupjkvvf] [] Decoding ctx of RPC function 'nb.get_base_image'
2021-03-01 12:31:17 run:95 [[DEBUG]] [TID=1rfupjkvvf] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.get_base_image'
2021-03-01 12:31:17 run:104 [[DEBUG]] [TID=1rfupjkvvf] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.get_base_image'
2021-03-01 12:31:17 run:114 [[INFO]] [TID=1rfupjkvvf] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'get_base_image()'
2021-03-01 12:31:17 run:83 [[DEBUG]] [TID=mxdqele2cx] [] Decoding ctx of RPC function 'nb.find_poddefault_labels_on_server'
2021-03-01 12:31:17 run:95 [[DEBUG]] [TID=mxdqele2cx] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.find_poddefault_labels_on_server'
2021-03-01 12:31:17 run:104 [[DEBUG]] [TID=mxdqele2cx] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.find_poddefault_labels_on_server'
2021-03-01 12:31:17 run:114 [[INFO]] [TID=mxdqele2cx] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'find_poddefault_labels_on_server()'
2021-03-01 12:31:17 nb:205 [[INFO]] [TID=mxdqele2cx] [/home/jovyan/Simple Notebook Pipeline.ipynb] Retrieving PodDefaults applied to server...
2021-03-01 12:31:17 nb:211 [[INFO]] [TID=mxdqele2cx] [/home/jovyan/Simple Notebook Pipeline.ipynb] Retrieved applied PodDefaults: []
2021-03-01 12:31:17 nb:215 [[INFO]] [TID=mxdqele2cx] [/home/jovyan/Simple Notebook Pipeline.ipynb] PodDefault labels applied on server:
2021-03-01 12:31:17 run:83 [[DEBUG]] [TID=br5u2s032y] [] Decoding ctx of RPC function 'kfp.list_experiments'
2021-03-01 12:31:17 run:95 [[DEBUG]] [TID=br5u2s032y] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'kfp.list_experiments'
2021-03-01 12:31:17 run:104 [[DEBUG]] [TID=br5u2s032y] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'kfp.list_experiments'
2021-03-01 12:31:17 run:114 [[INFO]] [TID=br5u2s032y] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'list_experiments()'
2021-03-01 12:31:35 run:83 [[DEBUG]] [TID=rnxs15j17o] [] Decoding ctx of RPC function 'nb.validate_notebook'
2021-03-01 12:31:35 run:95 [[DEBUG]] [TID=rnxs15j17o] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook'
2021-03-01 12:31:35 run:104 [[DEBUG]] [TID=rnxs15j17o] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.validate_notebook'
2021-03-01 12:31:35 run:114 [[INFO]] [TID=rnxs15j17o] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': 'new', 'name': 'asdfasdfasdfasdf'}, 'experiment_name': 'asdfasdfasdfasdf', 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'pipeline_description': 'asdfasdfasdfadsf', 'pipeline_name': 'newpipeline1', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': []})'
2021-03-01 12:31:35 run:83 [[DEBUG]] [TID=l5gn4ok577] [] Decoding ctx of RPC function 'nb.compile_notebook'
2021-03-01 12:31:35 run:95 [[DEBUG]] [TID=l5gn4ok577] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'nb.compile_notebook'
2021-03-01 12:31:35 run:104 [[DEBUG]] [TID=l5gn4ok577] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'nb.compile_notebook'
2021-03-01 12:31:35 run:114 [[INFO]] [TID=l5gn4ok577] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'compile_notebook(source_notebook_path=/home/jovyan/Simple Notebook Pipeline.ipynb, notebook_metadata_overrides={'autosnapshot': False, 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'experiment': {'id': 'new', 'name': 'asdfasdfasdfasdf'}, 'experiment_name': 'asdfasdfasdfasdf', 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'pipeline_description': 'asdfasdfasdfadsf', 'pipeline_name': 'newpipeline1', 'snapshot_volumes': False, 'steps_defaults': [], 'volume_access_mode': 'rwm', 'volumes': []}, debug=False)'
2021-03-01 12:31:35 core:146 [[DEBUG]] [TID=l5gn4ok577] [/home/jovyan/Simple Notebook Pipeline.ipynb] ------------- Kale Start Run -------------
2021-03-01 12:31:36 core:205 [[INFO]] [TID=l5gn4ok577] [/home/jovyan/Simple Notebook Pipeline.ipynb] Pipeline code saved at /home/jovyan/newpipeline1-goxmo.kale.py
2021-03-01 12:31:36 run:83 [[DEBUG]] [TID=fd0gzcnyl5] [] Decoding ctx of RPC function 'kfp.upload_pipeline'
2021-03-01 12:31:36 run:95 [[DEBUG]] [TID=fd0gzcnyl5] [/home/jovyan/Simple Notebook Pipeline.ipynb] Decoding kwargs of RPC function 'kfp.upload_pipeline'
2021-03-01 12:31:36 run:104 [[DEBUG]] [TID=fd0gzcnyl5] [/home/jovyan/Simple Notebook Pipeline.ipynb] Importing RPC function 'kfp.upload_pipeline'
2021-03-01 12:31:36 run:114 [[INFO]] [TID=fd0gzcnyl5] [/home/jovyan/Simple Notebook Pipeline.ipynb] Executing RPC function 'upload_pipeline(pipeline_package_path=newpipeline1-goxmo.pipeline.yaml, pipeline_metadata={'experiment_name': 'asdfasdfasdfasdf', 'pipeline_name': 'newpipeline1-goxmo', 'pipeline_description': 'asdfasdfasdfadsf', 'pipeline_args': '', 'pipeline_args_names': '', 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'volumes': [], 'abs_working_dir': '/home/jovyan', 'marshal_volume': True, 'marshal_path': '/marshal', 'snapshot_volumes': False, 'autosnapshot': False, 'steps_defaults': [], 'experiment': {'id': 'new', 'name': 'asdfasdfasdfasdf'}, 'katib_metadata': {'parameters': [], 'objective': {'objectiveMetricName': '', 'type': 'minimize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 12, 'maxFailedTrialCount': 3, 'parallelTrialCount': 3}, 'katib_run': False, 'volume_access_mode': 'rwm', 'parameters_names': [], 'parameters_types': [], 'parameters_values': []}, overwrite=False)'
2021-03-01 12:31:36 run:125 [[ERROR]] [TID=fd0gzcnyl5] [/home/jovyan/Simple Notebook Pipeline.ipynb] RPC function 'upload_pipeline' raised an unhandled exception
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run
result = func(request, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 96, in upload_pipeline
pipeline = client.upload_pipeline(pipeline_package_path, pipeline_name)
File "/home/jovyan/.local/lib/python3.6/site-packages/kfp/_client.py", line 881, in upload_pipeline
response = self._upload_api.upload_pipeline(pipeline_package_path, name=pipeline_name, description=description)
File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api/pipeline_upload_service_api.py", line 69, in upload_pipeline
return self.upload_pipeline_with_http_info(uploadfile, **kwargs) # noqa: E501
File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api/pipeline_upload_service_api.py", line 177, in upload_pipeline_with_http_info
collection_formats=collection_formats)
File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 369, in call_api
_preload_content, _request_timeout, _host)
File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 208, in __call_api
return_data = self.deserialize(response_data, response_type)
File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 280, in deserialize
return self.__deserialize(data, response_type)
File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 319, in __deserialize
return self.__deserialize_model(data, klass)
File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 658, in __deserialize_model
kwargs[attr] = self.__deserialize(value, attr_type)
File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 319, in __deserialize
return self.__deserialize_model(data, klass)
File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 658, in __deserialize_model
kwargs[attr] = self.__deserialize(value, attr_type)
File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 317, in __deserialize
return self.__deserialize_datetime(data)
File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 624, in __deserialize_datetime
return parse(string)
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 1374, in parse
return DEFAULTPARSER.parse(timestr, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 646, in parse
res, skipped_tokens = self._parse(timestr, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 725, in _parse
l = _timelex.split(timestr) # Splits the timestr into tokens
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 207, in split
return list(cls(s))
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 76, in init
'{itype}'.format(itype=instream.class.name))
TypeError: Parser must be a string or character stream, not dict
2021-03-01 12:39:10 run:83 [[DEBUG]] [TID=eq33czreem] [] Decoding ctx of RPC function 'nb.explore_notebook'
2021-03-01 12:39:10 run:95 [[DEBUG]] [TID=eq33czreem] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Decoding kwargs of RPC function 'nb.explore_notebook'
2021-03-01 12:39:10 run:104 [[DEBUG]] [TID=eq33czreem] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Importing RPC function 'nb.explore_notebook'
2021-03-01 12:39:10 run:114 [[INFO]] [TID=eq33czreem] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Executing RPC function 'explore_notebook(source_notebook_path=/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb)'
2021-03-01 12:39:10 run:83 [[DEBUG]] [TID=1b0dgoptqq] [] Decoding ctx of RPC function 'nb.get_base_image'
2021-03-01 12:39:10 run:95 [[DEBUG]] [TID=1b0dgoptqq] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Decoding kwargs of RPC function 'nb.get_base_image'
2021-03-01 12:39:10 run:104 [[DEBUG]] [TID=1b0dgoptqq] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Importing RPC function 'nb.get_base_image'
2021-03-01 12:39:10 run:114 [[INFO]] [TID=1b0dgoptqq] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Executing RPC function 'get_base_image()'
2021-03-01 12:39:10 run:83 [[DEBUG]] [TID=2iygvtbobb] [] Decoding ctx of RPC function 'nb.find_poddefault_labels_on_server'
2021-03-01 12:39:10 run:95 [[DEBUG]] [TID=2iygvtbobb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Decoding kwargs of RPC function 'nb.find_poddefault_labels_on_server'
2021-03-01 12:39:10 run:104 [[DEBUG]] [TID=2iygvtbobb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Importing RPC function 'nb.find_poddefault_labels_on_server'
2021-03-01 12:39:10 run:114 [[INFO]] [TID=2iygvtbobb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Executing RPC function 'find_poddefault_labels_on_server()'
2021-03-01 12:39:10 nb:205 [[INFO]] [TID=2iygvtbobb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Retrieving PodDefaults applied to server...
2021-03-01 12:39:11 nb:211 [[INFO]] [TID=2iygvtbobb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Retrieved applied PodDefaults: []
2021-03-01 12:39:11 nb:215 [[INFO]] [TID=2iygvtbobb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] PodDefault labels applied on server:
2021-03-01 12:39:11 run:83 [[DEBUG]] [TID=pfq35lwh69] [] Decoding ctx of RPC function 'kfp.list_experiments'
2021-03-01 12:39:11 run:95 [[DEBUG]] [TID=pfq35lwh69] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Decoding kwargs of RPC function 'kfp.list_experiments'
2021-03-01 12:39:11 run:104 [[DEBUG]] [TID=pfq35lwh69] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Importing RPC function 'kfp.list_experiments'
2021-03-01 12:39:11 run:114 [[INFO]] [TID=pfq35lwh69] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Executing RPC function 'list_experiments()'
2021-03-01 12:39:42 run:83 [[DEBUG]] [TID=kujjlftjrv] [] Decoding ctx of RPC function 'nb.validate_notebook'
2021-03-01 12:39:42 run:95 [[DEBUG]] [TID=kujjlftjrv] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Decoding kwargs of RPC function 'nb.validate_notebook'
2021-03-01 12:39:42 run:104 [[DEBUG]] [TID=kujjlftjrv] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Importing RPC function 'nb.validate_notebook'
2021-03-01 12:39:42 run:114 [[INFO]] [TID=kujjlftjrv] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Executing RPC function 'validate_notebook(source_notebook_path=/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb, notebook_metadata_overrides={'experiment': {'id': 'new', 'name': 'dog-breed-katib'}, 'experiment_name': 'dog-breed-katib', 'katib_metadata': {'parameters': [{'feasibleSpace': {'max': '2100', 'min': '128', 'step': '128'}, 'name': 'nodes_number', 'parameterType': 'double'}, {'feasibleSpace': {'list': ['0.0001', '0.0005', '0.001', '0.01', '0.1', '0.005', '0.05', '0.5']}, 'name': 'lr', 'parameterType': 'categorical'}], 'objective': {'additionalMetricNames': [], 'goal': 90, 'objectiveMetricName': 'test-accuracy-resnet', 'type': 'maximize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 200, 'maxFailedTrialCount': 3, 'parallelTrialCount': 1}, 'katib_run': False, 'pipeline_description': '', 'pipeline_name': 'dog-breed', 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'volumes': [], 'autosnapshot': False, 'snapshot_volumes': False, 'steps_defaults': []})'
2021-03-01 12:39:42 run:83 [[DEBUG]] [TID=6ozisotmwb] [] Decoding ctx of RPC function 'nb.compile_notebook'
2021-03-01 12:39:42 run:95 [[DEBUG]] [TID=6ozisotmwb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Decoding kwargs of RPC function 'nb.compile_notebook'
2021-03-01 12:39:42 run:104 [[DEBUG]] [TID=6ozisotmwb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Importing RPC function 'nb.compile_notebook'
2021-03-01 12:39:42 run:114 [[INFO]] [TID=6ozisotmwb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Executing RPC function 'compile_notebook(source_notebook_path=/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb, notebook_metadata_overrides={'experiment': {'id': 'new', 'name': 'dog-breed-katib'}, 'experiment_name': 'dog-breed-katib', 'katib_metadata': {'parameters': [{'feasibleSpace': {'max': '2100', 'min': '128', 'step': '128'}, 'name': 'nodes_number', 'parameterType': 'double'}, {'feasibleSpace': {'list': ['0.0001', '0.0005', '0.001', '0.01', '0.1', '0.005', '0.05', '0.5']}, 'name': 'lr', 'parameterType': 'categorical'}], 'objective': {'additionalMetricNames': [], 'goal': 90, 'objectiveMetricName': 'test-accuracy-resnet', 'type': 'maximize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 200, 'maxFailedTrialCount': 3, 'parallelTrialCount': 1}, 'katib_run': False, 'pipeline_description': '', 'pipeline_name': 'dog-breed', 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'volumes': [], 'autosnapshot': False, 'snapshot_volumes': False, 'steps_defaults': []}, debug=False)'
2021-03-01 12:39:42 core:146 [[DEBUG]] [TID=6ozisotmwb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] ------------- Kale Start Run -------------
2021-03-01 12:39:44 core:205 [[INFO]] [TID=6ozisotmwb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Pipeline code saved at /home/jovyan/kale/examples/dog-breed-classification/dog-breed-a9bpu.kale.py
2021-03-01 12:39:44 driver:124 [[INFO]] Generating grammar tables from /usr/lib/python3.6/lib2to3/Grammar.txt
2021-03-01 12:39:44 driver:124 [[INFO]] Generating grammar tables from /usr/lib/python3.6/lib2to3/PatternGrammar.txt
2021-03-01 12:39:45 run:83 [[DEBUG]] [TID=n8twyqkq8p] [] Decoding ctx of RPC function 'kfp.upload_pipeline'
2021-03-01 12:39:45 run:95 [[DEBUG]] [TID=n8twyqkq8p] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Decoding kwargs of RPC function 'kfp.upload_pipeline'
2021-03-01 12:39:45 run:104 [[DEBUG]] [TID=n8twyqkq8p] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Importing RPC function 'kfp.upload_pipeline'
2021-03-01 12:39:45 run:114 [[INFO]] [TID=n8twyqkq8p] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Executing RPC function 'upload_pipeline(pipeline_package_path=kale/examples/dog-breed-classification/dog-breed-a9bpu.pipeline.yaml, pipeline_metadata={'experiment_name': 'dog-breed-katib', 'pipeline_name': 'dog-breed-a9bpu', 'pipeline_description': '', 'pipeline_args': '', 'pipeline_args_names': '', 'docker_image': 'gcr.io/arrikto/jupyter-kale:v0.5.0-47-g2427cc9', 'volumes': [], 'abs_working_dir': '/home/jovyan/kale/examples/dog-breed-classification', 'marshal_volume': True, 'marshal_path': '/marshal', 'snapshot_volumes': False, 'autosnapshot': False, 'steps_defaults': [], 'experiment': {'id': 'new', 'name': 'dog-breed-katib'}, 'katib_metadata': {'parameters': [{'feasibleSpace': {'max': '2100', 'min': '128', 'step': '128'}, 'name': 'nodes_number', 'parameterType': 'double'}, {'feasibleSpace': {'list': ['0.0001', '0.0005', '0.001', '0.01', '0.1', '0.005', '0.05', '0.5']}, 'name': 'lr', 'parameterType': 'categorical'}], 'objective': {'additionalMetricNames': [], 'goal': 90, 'objectiveMetricName': 'test-accuracy-resnet', 'type': 'maximize'}, 'algorithm': {'algorithmName': 'grid'}, 'maxTrialCount': 200, 'maxFailedTrialCount': 3, 'parallelTrialCount': 1}, 'katib_run': False, 'parameters_names': ['learning_rate', 'nodes_number'], 'parameters_types': ['float', 'int'], 'parameters_values': [0.0001, 256]}, overwrite=False)'
2021-03-01 12:39:45 run:125 [[ERROR]] [TID=n8twyqkq8p] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] RPC function 'upload_pipeline' raised an unhandled exception
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run
result = func(request, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 96, in upload_pipeline
pipeline = client.upload_pipeline(pipeline_package_path, pipeline_name)
File "/home/jovyan/.local/lib/python3.6/site-packages/kfp/_client.py", line 881, in upload_pipeline
response = self._upload_api.upload_pipeline(pipeline_package_path, name=pipeline_name, description=description)
File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api/pipeline_upload_service_api.py", line 69, in upload_pipeline
return self.upload_pipeline_with_http_info(uploadfile, **kwargs) # noqa: E501
File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api/pipeline_upload_service_api.py", line 177, in upload_pipeline_with_http_info
collection_formats=collection_formats)
File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 369, in call_api
_preload_content, _request_timeout, _host)
File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 208, in __call_api
return_data = self.deserialize(response_data, response_type)
File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 280, in deserialize
return self.__deserialize(data, response_type)
File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 319, in __deserialize
return self.__deserialize_model(data, klass)
File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 658, in __deserialize_model
kwargs[attr] = self.__deserialize(value, attr_type)
File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 319, in __deserialize
return self.__deserialize_model(data, klass)
File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 658, in __deserialize_model
kwargs[attr] = self.__deserialize(value, attr_type)
File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 317, in __deserialize
return self.__deserialize_datetime(data)
File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 624, in __deserialize_datetime
return parse(string)
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 1374, in parse
return DEFAULTPARSER.parse(timestr, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 646, in parse
res, skipped_tokens = self._parse(timestr, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 725, in _parse
l = _timelex.split(timestr) # Splits the timestr into tokens
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 207, in split
return list(cls(s))
File "/usr/local/lib/python3.6/dist-packages/dateutil/parser/_parser.py", line 76, in init
'{itype}'.format(itype=instream.class.name))
TypeError: Parser must be a string or character stream, not dict
2021-03-01 13:17:00 run:83 [[DEBUG]] [TID=qhlnogi3bb] [] Decoding ctx of RPC function 'rok.check_rok_availability'
2021-03-01 13:17:00 run:95 [[DEBUG]] [TID=qhlnogi3bb] [] Decoding kwargs of RPC function 'rok.check_rok_availability'
2021-03-01 13:17:00 run:104 [[DEBUG]] [TID=qhlnogi3bb] [] Importing RPC function 'rok.check_rok_availability'
2021-03-01 13:17:02 run:114 [[INFO]] [TID=qhlnogi3bb] [] Executing RPC function 'check_rok_availability()'
2021-03-01 13:17:02 rok:123 [[ERROR]] [TID=qhlnogi3bb] [] Failed to initialize RokClient
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 117, in check_rok_availability
rok = _get_client()
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 42, in _get_client
_client = RokClient()
File "build/bdist.linux-x86_64/egg/rok_gw_client/client.py", line 175, in init
ValueError: A Rok API URL must be provided either via the url' argument or the ROK_GW_URL' environment variable.
2021-03-01 13:17:02 run:120 [[ERROR]] [TID=qhlnogi3bb] [] RPC function 'check_rok_availability' raised an RPCError
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 117, in check_rok_availability
rok = _get_client()
File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 42, in _get_client
_client = RokClient()
File "build/bdist.linux-x86_64/egg/rok_gw_client/client.py", line 175, in init
ValueError: A Rok API URL must be provided either via the url' argument or the ROK_GW_URL' environment variable.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run result = func(request, **kwargs) File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 126, in check_rok_availability trans_id=request.trans_id) kale.rpc.errors.RPCServiceUnavailableError 2021-03-01 13:17:04 run:83 [[DEBUG]] [TID=573x0u1g4z] [] Decoding ctx of RPC function 'nb.resume_notebook_path' 2021-03-01 13:17:04 run:95 [[DEBUG]] [TID=573x0u1g4z] [] Decoding kwargs of RPC function 'nb.resume_notebook_path' 2021-03-01 13:17:04 run:104 [[DEBUG]] [TID=573x0u1g4z] [] Importing RPC function 'nb.resume_notebook_path' 2021-03-01 13:17:05 run:114 [[INFO]] [TID=573x0u1g4z] [] Executing RPC function 'resume_notebook_path(server_root=/home/jovyan)' 2021-03-01 13:17:06 run:83 [[DEBUG]] [TID=paouw6e22o] [] Decoding ctx of RPC function 'nb.explore_notebook' 2021-03-01 13:17:06 run:95 [[DEBUG]] [TID=paouw6e22o] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Decoding kwargs of RPC function 'nb.explore_notebook' 2021-03-01 13:17:06 run:104 [[DEBUG]] [TID=paouw6e22o] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Importing RPC function 'nb.explore_notebook' 2021-03-01 13:17:06 run:114 [[INFO]] [TID=paouw6e22o] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Executing RPC function 'explore_notebook(source_notebook_path=/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb)' 2021-03-01 13:17:07 run:83 [[DEBUG]] [TID=4cheyuki97] [] Decoding ctx of RPC function 'nb.get_base_image' 2021-03-01 13:17:07 run:95 [[DEBUG]] [TID=4cheyuki97] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Decoding kwargs of RPC function 'nb.get_base_image' 2021-03-01 13:17:07 run:104 [[DEBUG]] [TID=4cheyuki97] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Importing RPC function 'nb.get_base_image' 2021-03-01 13:17:07 run:114 [[INFO]] [TID=4cheyuki97] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Executing RPC function 'get_base_image()' 2021-03-01 13:17:07 run:83 [[DEBUG]] [TID=vg4gmhe3yb] [] Decoding ctx of RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 13:17:07 run:95 [[DEBUG]] [TID=vg4gmhe3yb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Decoding kwargs of RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 13:17:07 run:104 [[DEBUG]] [TID=vg4gmhe3yb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Importing RPC function 'nb.find_poddefault_labels_on_server' 2021-03-01 13:17:07 run:114 [[INFO]] [TID=vg4gmhe3yb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Executing RPC function 'find_poddefault_labels_on_server()' 2021-03-01 13:17:07 nb:205 [[INFO]] [TID=vg4gmhe3yb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Retrieving PodDefaults applied to server... 2021-03-01 13:17:07 nb:211 [[INFO]] [TID=vg4gmhe3yb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Retrieved applied PodDefaults: [] 2021-03-01 13:17:07 nb:215 [[INFO]] [TID=vg4gmhe3yb] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] PodDefault labels applied on server: 2021-03-01 13:17:08 run:83 [[DEBUG]] [TID=hkaedck5w5] [] Decoding ctx of RPC function 'kfp.list_experiments' 2021-03-01 13:17:08 run:95 [[DEBUG]] [TID=hkaedck5w5] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Decoding kwargs of RPC function 'kfp.list_experiments' 2021-03-01 13:17:08 run:104 [[DEBUG]] [TID=hkaedck5w5] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Importing RPC function 'kfp.list_experiments' 2021-03-01 13:17:08 run:114 [[INFO]] [TID=hkaedck5w5] [/home/jovyan/kale/examples/dog-breed-classification/dog-breed.ipynb] Executing RPC function 'list_experiments()'
Hi, could you also provide
- the KFP API server deployment version (deployment
ml-pipeline) - the KFP SDK and
kfp-server-apiversions you have installed in the image
This sounds like an old KFP API server bug: https://github.com/kubeflow/pipelines/issues/2764, https://github.com/kubeflow/pipelines/issues/3441, https://github.com/kubeflow/kubeflow/issues/4830
