kale icon indicating copy to clipboard operation
kale copied to clipboard

Published 20 hours ago • verified publisher icon kubeflow-kale

TypeError: expected string or bytes-like object

Open frankmanbb opened this issue 4 years ago • 2 comments

I installed kubeflow 1.2 within my baremetal k8s server and it works very good with the original notebook. now I've found kale and want to try it. I use the custom image gcr.io/arrikto/jupyter-kale:4a5d7e63-9f74f267, but encountered issue:

2021-01-06 08:58:29 run:83 [[DEBUG]] [TID=rn5borsk9d] [] Decoding ctx of RPC function 'rok.check_rok_availability' 2021-01-06 08:58:29 run:95 [[DEBUG]] [TID=rn5borsk9d] [] Decoding kwargs of RPC function 'rok.check_rok_availability' 2021-01-06 08:58:29 run:104 [[DEBUG]] [TID=rn5borsk9d] [] Importing RPC function 'rok.check_rok_availability' 2021-01-06 08:58:29 run:114 [[INFO]] [TID=rn5borsk9d] [] Executing RPC function 'check_rok_availability()' 2021-01-06 08:58:29 rok:50 [[ERROR]] [TID=rn5borsk9d] [] Failed to initialize RokClient Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 44, in check_rok_availability rok = rokutils.get_client() File "/usr/local/lib/python3.6/dist-packages/kale/common/rokutils.py", line 49, in get_client _client = RokClient() File "build/bdist.linux-x86_64/egg/rok_gw_client/client.py", line 175, in init ValueError: A Rok API URL must be provided either via the url' argument or the ROK_GW_URL' environment variable. 2021-01-06 08:58:29 run:120 [[ERROR]] [TID=rn5borsk9d] [] RPC function 'check_rok_availability' raised an RPCError Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 44, in check_rok_availability rok = rokutils.get_client() File "/usr/local/lib/python3.6/dist-packages/kale/common/rokutils.py", line 49, in get_client _client = RokClient() File "build/bdist.linux-x86_64/egg/rok_gw_client/client.py", line 175, in init ValueError: A Rok API URL must be provided either via the url' argument or the ROK_GW_URL' environment variable.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run result = func(request, **kwargs) File "/usr/local/lib/python3.6/dist-packages/kale/rpc/rok.py", line 53, in check_rok_availability trans_id=request.trans_id) kale.rpc.errors.RPCServiceUnavailableError 2021-01-06 08:58:30 run:83 [[DEBUG]] [TID=ppq88glavz] [] Decoding ctx of RPC function 'nb.resume_notebook_path' 2021-01-06 08:58:30 run:95 [[DEBUG]] [TID=ppq88glavz] [] Decoding kwargs of RPC function 'nb.resume_notebook_path' 2021-01-06 08:58:30 run:104 [[DEBUG]] [TID=ppq88glavz] [] Importing RPC function 'nb.resume_notebook_path' 2021-01-06 08:58:30 run:114 [[INFO]] [TID=ppq88glavz] [] Executing RPC function 'resume_notebook_path(server_root=/home/jovyan)' 2021-01-06 08:58:30 run:83 [[DEBUG]] [TID=swjghbxvyl] [] Decoding ctx of RPC function 'nb.get_namespace' 2021-01-06 08:58:30 run:95 [[DEBUG]] [TID=swjghbxvyl] [/home/jovyan/Untitled.ipynb] Decoding kwargs of RPC function 'nb.get_namespace' 2021-01-06 08:58:30 run:104 [[DEBUG]] [TID=swjghbxvyl] [/home/jovyan/Untitled.ipynb] Importing RPC function 'nb.get_namespace' 2021-01-06 08:58:30 run:114 [[INFO]] [TID=swjghbxvyl] [/home/jovyan/Untitled.ipynb] Executing RPC function 'get_namespace()' 2021-01-06 08:58:30 nb:233 [[INFO]] [TID=swjghbxvyl] [/home/jovyan/Untitled.ipynb] Retrieving notebook's namespace... 2021-01-06 08:58:30 nb:235 [[INFO]] [TID=swjghbxvyl] [/home/jovyan/Untitled.ipynb] Notebook's namespace is 'kfadmin' 2021-01-06 08:58:30 run:83 [[DEBUG]] [TID=oj69fob9o8] [] Decoding ctx of RPC function 'nb.explore_notebook' 2021-01-06 08:58:30 run:95 [[DEBUG]] [TID=oj69fob9o8] [/home/jovyan/Untitled.ipynb] Decoding kwargs of RPC function 'nb.explore_notebook' 2021-01-06 08:58:30 run:104 [[DEBUG]] [TID=oj69fob9o8] [/home/jovyan/Untitled.ipynb] Importing RPC function 'nb.explore_notebook' 2021-01-06 08:58:30 run:114 [[INFO]] [TID=oj69fob9o8] [/home/jovyan/Untitled.ipynb] Executing RPC function 'explore_notebook(source_notebook_path=/home/jovyan/Untitled.ipynb)' 2021-01-06 08:58:31 run:83 [[DEBUG]] [TID=zq99z018pe] [] Decoding ctx of RPC function 'nb.get_base_image' 2021-01-06 08:58:31 run:95 [[DEBUG]] [TID=zq99z018pe] [/home/jovyan/Untitled.ipynb] Decoding kwargs of RPC function 'nb.get_base_image' 2021-01-06 08:58:31 run:104 [[DEBUG]] [TID=zq99z018pe] [/home/jovyan/Untitled.ipynb] Importing RPC function 'nb.get_base_image' 2021-01-06 08:58:31 run:114 [[INFO]] [TID=zq99z018pe] [/home/jovyan/Untitled.ipynb] Executing RPC function 'get_base_image()' 2021-01-06 08:58:31 run:83 [[DEBUG]] [TID=haiwiqyejz] [] Decoding ctx of RPC function 'nb.find_poddefault_labels_on_server' 2021-01-06 08:58:31 run:95 [[DEBUG]] [TID=haiwiqyejz] [/home/jovyan/Untitled.ipynb] Decoding kwargs of RPC function 'nb.find_poddefault_labels_on_server' 2021-01-06 08:58:31 run:104 [[DEBUG]] [TID=haiwiqyejz] [/home/jovyan/Untitled.ipynb] Importing RPC function 'nb.find_poddefault_labels_on_server' 2021-01-06 08:58:31 run:114 [[INFO]] [TID=haiwiqyejz] [/home/jovyan/Untitled.ipynb] Executing RPC function 'find_poddefault_labels_on_server()' 2021-01-06 08:58:31 nb:217 [[INFO]] [TID=haiwiqyejz] [/home/jovyan/Untitled.ipynb] Retrieving PodDefaults applied to server... 2021-01-06 08:58:31 nb:223 [[INFO]] [TID=haiwiqyejz] [/home/jovyan/Untitled.ipynb] Retrieved applied PodDefaults: [] 2021-01-06 08:58:31 nb:227 [[INFO]] [TID=haiwiqyejz] [/home/jovyan/Untitled.ipynb] PodDefault labels applied on server: 2021-01-06 08:58:31 run:83 [[DEBUG]] [TID=gileovsfr8] [] Decoding ctx of RPC function 'kfp.list_experiments' 2021-01-06 08:58:31 run:95 [[DEBUG]] [TID=gileovsfr8] [/home/jovyan/Untitled.ipynb] Decoding kwargs of RPC function 'kfp.list_experiments' 2021-01-06 08:58:31 run:104 [[DEBUG]] [TID=gileovsfr8] [/home/jovyan/Untitled.ipynb] Importing RPC function 'kfp.list_experiments' 2021-01-06 08:58:31 run:114 [[INFO]] [TID=gileovsfr8] [/home/jovyan/Untitled.ipynb] Executing RPC function 'list_experiments()' 2021-01-06 08:58:31 run:125 [[ERROR]] [TID=gileovsfr8] [/home/jovyan/Untitled.ipynb] RPC function 'list_experiments' raised an unhandled exception Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/kale/rpc/run.py", line 116, in run result = func(request, **kwargs) File "/usr/local/lib/python3.6/dist-packages/kale/rpc/kfp.py", line 30, in list_experiments for e in c.list_experiments().experiments or []] File "/usr/local/lib/python3.6/dist-packages/kfp/_client.py", line 326, in list_experiments resource_reference_key_id=namespace) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api/experiment_service_api.py", line 581, in list_experiment return self.list_experiment_with_http_info(**kwargs) # noqa: E501 File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api/experiment_service_api.py", line 696, in list_experiment_with_http_info collection_formats=collection_formats) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 383, in call_api _preload_content, _request_timeout, _host) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 199, in __call_api _request_timeout=_request_timeout) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/api_client.py", line 407, in request headers=headers) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/rest.py", line 248, in GET query_params=query_params) File "/usr/local/lib/python3.6/dist-packages/kfp_server_api/rest.py", line 226, in request headers=headers) File "/usr/local/lib/python3.6/dist-packages/urllib3/request.py", line 75, in request method, url, fields=fields, headers=headers, **urlopen_kw File "/usr/local/lib/python3.6/dist-packages/urllib3/request.py", line 96, in request_encode_url return self.urlopen(method, url, **extra_kw) File "/usr/local/lib/python3.6/dist-packages/urllib3/poolmanager.py", line 375, in urlopen response = conn.urlopen(method, u.request_uri, **kw) File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 706, in urlopen chunked=chunked, File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 394, in _make_request conn.request(method, url, **httplib_request_kw) File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 234, in request super(HTTPConnection, self).request(method, url, body=body, headers=headers) File "/usr/lib/python3.6/http/client.py", line 1281, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3.6/http/client.py", line 1322, in _send_request self.putheader(hdr, value) File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 219, in putheader _HTTPConnection.putheader(self, header, *values) File "/usr/lib/python3.6/http/client.py", line 1258, in putheader if _is_illegal_header_value(values[i]): TypeError: expected string or bytes-like object

do we know what is wrong? Thanks!

frankmanbb avatar Jan 06 '21 09:01 frankmanbb 

if _is_illegal_header_value(values[i]):

It's probably your authorization header. You can confirm by opening up one of the files in your callstack and adding some logging.

Do you have an env var pointing at ML_PIPELINE_SA_TOKEN_PATH? The poddefaults in this comment might help: https://github.com/kubeflow/katib/issues/1454#issuecomment-791384896

EDIT: This is not expected to solve your original issue. The original issue is that ROK_GW_URL is not set. To check if rok is running, Kale probably tried to execute an rpc which required the auth header. So even if you fix the latter you will hit the former, and viceversea.

prashanthb-ai avatar Mar 07 '21 14:03 prashanthb-ai

@frankmanbb have you solved it now?

Bowen0729 avatar Jun 11 '21 03:06 Bowen0729