Why kedro-viz needs connection to mlflow to open?

[foxale]

trafficstars

Description

I want to run kedro-viz without connecting to the mlflow tracking server instance.

Steps to Reproduce

My mlflow tracking server is on a separate machine and I'm not connected to it. I run kedro viz and see the following:

2022-10-07 18:03:36,900 - urllib3.connectionpool - WARNING - Retrying (Retry(total=4, connect=4, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x125481160>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/2.0/mlflow/experiments/get-by-name?experiment_name=test-experiment

[foxale]

I also need to have all Python libraries installed. But.. why?

[tynandebold]

Hey @foxale, thank you for the bug report.

Kedro-Viz doesn't need a connection to MLflow to open. I'm not sure what your setup is, though would guess that your mlflow tracking server's process (or some other process on your machine) is somehow conflicting with the Kedro-Viz processes.

Do you have anything running on ports 4141 or 4142?

[foxale]

I don't have anything running on those ports. The error message does not appear when using other kedro commands like kedro jupyter lab, kedro lint, etc. However, it appears while running kedro run (which is expected, as kedro run logs stuff to mlflow).

Here's a full console log:

2022-10-15 20:02:15,859 - urllib3.connectionpool - WARNING - Retrying (Retry(total=4, connect=4, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1356a8160>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/2.0/mlflow/experiments/get-by-name?experiment_name=update_conenction_strings
2022-10-15 20:02:19,872 - urllib3.connectionpool - WARNING - Retrying (Retry(total=3, connect=3, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1356a85b0>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/2.0/mlflow/experiments/get-by-name?experiment_name=update_conenction_strings
2022-10-15 20:02:27,877 - urllib3.connectionpool - WARNING - Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1356a9ac0>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/2.0/mlflow/experiments/get-by-name?experiment_name=update_conenction_strings
2022-10-15 20:02:43,883 - urllib3.connectionpool - WARNING - Retrying (Retry(total=1, connect=1, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1356a9f70>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/2.0/mlflow/experiments/get-by-name?experiment_name=update_conenction_strings
2022-10-15 20:03:15,891 - urllib3.connectionpool - WARNING - Retrying (Retry(total=0, connect=0, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1356a9af0>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/2.0/mlflow/experiments/get-by-name?experiment_name=update_conenction_strings
2022-10-15 20:03:15,899 - urllib3.connectionpool - WARNING - Retrying (Retry(total=4, connect=4, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x135708550>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/2.0/mlflow/experiments/list?view_type=ALL
2022-10-15 20:03:19,905 - urllib3.connectionpool - WARNING - Retrying (Retry(total=3, connect=3, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1356c0fa0>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/2.0/mlflow/experiments/list?view_type=ALL
2022-10-15 20:03:27,907 - urllib3.connectionpool - WARNING - Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1356c0b50>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/2.0/mlflow/experiments/list?view_type=ALL
2022-10-15 20:03:43,909 - urllib3.connectionpool - WARNING - Retrying (Retry(total=1, connect=1, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1356c0e20>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/2.0/mlflow/experiments/list?view_type=ALL
2022-10-15 20:04:15,916 - urllib3.connectionpool - WARNING - Retrying (Retry(total=0, connect=0, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1356a9580>: Failed to establish a new connection: [Errno 61] Connection refused')': /api/2.0/mlflow/experiments/list?view_type=ALL
Traceback (most recent call last):
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/urllib3/connection.py", line 159, in _new_conn
    conn = connection.create_connection(
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/http/client.py", line 950, in send
    self.connect()
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/urllib3/connection.py", line 171, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x1356a97f0>: Failed to establish a new connection: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/urllib3/connectionpool.py", line 754, in urlopen
    return self.urlopen(
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/urllib3/connectionpool.py", line 754, in urlopen
    return self.urlopen(
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/urllib3/connectionpool.py", line 754, in urlopen
    return self.urlopen(
  [Previous line repeated 2 more times]
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/urllib3/util/retry.py", line 446, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=5001): Max retries exceeded with url: /api/2.0/mlflow/experiments/get-by-name?experiment_name=update_conenction_strings (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1356a97f0>: Failed to establish a new connection: [Errno 61] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/mlflow/utils/rest_utils.py", line 151, in http_request
    return _get_http_response_with_retries(
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/mlflow/utils/rest_utils.py", line 91, in _get_http_response_with_retries
    return session.request(method, url, **kwargs)
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=5001): Max retries exceeded with url: /api/2.0/mlflow/experiments/get-by-name?experiment_name=update_conenction_strings (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1356a97f0>: Failed to establish a new connection: [Errno 61] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/mlflow/store/tracking/rest_store.py", line 285, in get_experiment_by_name
    response_proto = self._call_endpoint(GetExperimentByName, req_body)
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/mlflow/store/tracking/rest_store.py", line 56, in _call_endpoint
    return call_endpoint(self.get_host_creds(), endpoint, method, json_body, response_proto)
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/mlflow/utils/rest_utils.py", line 249, in call_endpoint
    response = http_request(
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/mlflow/utils/rest_utils.py", line 163, in http_request
    raise MlflowException("API request to %s failed with exception %s" % (url, e))
mlflow.exceptions.MlflowException: API request to http://localhost:5001/api/2.0/mlflow/experiments/get-by-name failed with exception HTTPConnectionPool(host='localhost', port=5001): Max retries exceeded with url: /api/2.0/mlflow/experiments/get-by-name?experiment_name=update_conenction_strings (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1356a97f0>: Failed to establish a new connection: [Errno 61] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/urllib3/connection.py", line 159, in _new_conn
    conn = connection.create_connection(
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/http/client.py", line 950, in send
    self.connect()
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/urllib3/connection.py", line 171, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x1356a8790>: Failed to establish a new connection: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/urllib3/connectionpool.py", line 754, in urlopen
    return self.urlopen(
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/urllib3/connectionpool.py", line 754, in urlopen
    return self.urlopen(
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/urllib3/connectionpool.py", line 754, in urlopen
    return self.urlopen(
  [Previous line repeated 2 more times]
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/urllib3/util/retry.py", line 446, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=5001): Max retries exceeded with url: /api/2.0/mlflow/experiments/list?view_type=ALL (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1356a8790>: Failed to establish a new connection: [Errno 61] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/mlflow/utils/rest_utils.py", line 151, in http_request
    return _get_http_response_with_retries(
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/mlflow/utils/rest_utils.py", line 91, in _get_http_response_with_retries
    return session.request(method, url, **kwargs)
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=5001): Max retries exceeded with url: /api/2.0/mlflow/experiments/list?view_type=ALL (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1356a8790>: Failed to establish a new connection: [Errno 61] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/kedro_viz/launchers/cli.py", line 131, in viz
    run_server(**run_server_kwargs)
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/kedro_viz/server.py", line 81, in run_server
    catalog, pipelines, session_store_location = kedro_data_loader.load_data(
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/kedro_viz/integrations/kedro/data_loader.py", line 79, in load_data
    context = session.load_context()
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/kedro/framework/session/session.py", line 261, in load_context
    self._hook_manager.hook.after_context_created(  # pylint: disable=no-member
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/kedro_mlflow/framework/hooks/pipeline_hook.py", line 42, in after_context_created
    mlflow_config.setup(context)  # setup global mlflow configuration
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/kedro_mlflow/config/kedro_mlflow_config.py", line 124, in setup
    self._set_experiment()
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/kedro_mlflow/config/kedro_mlflow_config.py", line 141, in _set_experiment
    mlflow_experiment = self.server._mlflow_client.get_experiment_by_name(
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/mlflow/tracking/client.py", line 462, in get_experiment_by_name
    return self._tracking_client.get_experiment_by_name(name)
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/mlflow/tracking/_tracking_service/client.py", line 162, in get_experiment_by_name
    return self.store.get_experiment_by_name(name)
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/mlflow/store/tracking/rest_store.py", line 297, in get_experiment_by_name
    for experiment in self.list_experiments(ViewType.ALL):
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/mlflow/store/tracking/rest_store.py", line 77, in list_experiments
    response_proto = self._call_endpoint(ListExperiments, req_body)
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/mlflow/store/tracking/rest_store.py", line 56, in _call_endpoint
    return call_endpoint(self.get_host_creds(), endpoint, method, json_body, response_proto)
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/mlflow/utils/rest_utils.py", line 249, in call_endpoint
    response = http_request(
  File "/Users/plis003/.pyenv/versions/3.9.2/lib/python3.9/site-packages/mlflow/utils/rest_utils.py", line 163, in http_request
    raise MlflowException("API request to %s failed with exception %s" % (url, e))
mlflow.exceptions.MlflowException: API request to http://localhost:5001/api/2.0/mlflow/experiments/list failed with exception HTTPConnectionPool(host='localhost', port=5001): Max retries exceeded with url: /api/2.0/mlflow/experiments/list?view_type=ALL (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1356a8790>: Failed to establish a new connection: [Errno 61] Connection refused'))
kedro.framework.cli.utils.KedroCliError: API request to http://localhost:5001/api/2.0/mlflow/experiments/list failed with exception HTTPConnectionPool(host='localhost', port=5001): Max retries exceeded with url: /api/2.0/mlflow/experiments/list?view_type=ALL (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1356a8790>: Failed to establish a new connection: [Errno 61] Connection refused'))
Run with --verbose to see the full exception
Error: API request to http://localhost:5001/api/2.0/mlflow/experiments/list failed with exception HTTPConnectionPool(host='localhost', port=5001): Max retries exceeded with url: /api/2.0/mlflow/experiments/list?view_type=ALL (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1356a8790>: Failed to establish a new connection: [Errno 61] Connection refused'))```

[foxale]

After a quick debug session, it seems the problem is with run_server function, specifically this line:

        catalog, pipelines, session_store_location = kedro_data_loader.load_data(
            path, env, extra_params
        )

[tynandebold]

Thanks for the detailed stacktrace @foxale. The reason for this is that when you run Kedro-Viz all plugin hooks get executed, and if you have the kedro_mlflow plugin installed, it's expecting that plugin to be running on that port.

You'll need to disable this plugin in your own setup in order to run Kedro-Viz successfully. I think they have some settings where you can disable it.

Let me know if you have any further issues.

[foxale]

Thanks for the proposed solution, that makes sense.

Isn't that still a bug though? Like, kedro-viz is completely independent from kedro-mlflow, so why would you need to change the source code of you project (=disable the kedro-mlflow plugin) in order to be able to run kedro-viz? I'm not even sure whether that's something that should be fixed on kedro's, kedro-mflow's or kedro-viz's side, but purely from a user perspective it doesn't make much sense.

[tynandebold]

We don't support or maintain that specific plugin, so if it's a bug I'd say that it wouldn't be on us to solve.

[foxale]

So I had a while to think about it, and I think it's still (at least just a little bit) on kedro-viz. This package is related only to "core" kedro pipelines and nothing else, why would it have to depend on the correctness of other plugins? As a user, I would want to examine my kedro pipelines in kedro viz no matter how broken my plugins or network connection is.

Same goes with python packages used inside pipelines — I don't need them to generate the kedro-viz's webpage, so would why kedro-viz throw an exception when I don't have all them installed?

[foxale]

@tynandebold bump

[tynandebold]

@foxale see this comment.