Possibility to use only the python from docker image

[Honzys]

Hello,

I would like to ask you, if it's possible to use only the python venv from the docker image passed as argument to clearml-task without installing other python packages?

I have a docker image, that is fully compatible with my Machine Learning training. I have a really ugly hack in the python entrypoint to get rid of the new installed python and to set correct PYTHONPATH, so it uses correct python environment, that is already prepared in the image.

It would be great to have some option to disable all further installation / preparation of python evnrinoment and use the prepared env as it is. Is it possible now? Or are you thinking about adding this feature in som future version?

Thanks in advance :)

[jkhenning]

Hi @Honzys,

You can use the CLEARML_AGENT_SKIP_PIP_VENV_INSTALL environment variable to specify the python executable inside the docker image (this env var will be picked up by the clearml agent running inside the docker container), will that do?

[Honzys]

Thank you for your response!

Thanks for the pointers, but after setting the evnironment variable I am getting following error.

Environment setup completed successfully
Starting Task Execution:
[Errno 2] No such file or directory
Leaving process id 26

I tried changing the agent.python_binary settings but without success.

Sadly I don't know which file or directory is missing, because it is not in the log.

[jkhenning]

How did you pass the env var, and what value did you use?

[Honzys]

I was searching about the usage and values for the environment, I passed CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=1 clearml daemon ... accoring to - https://githubmemory.com/repo/allegroai/clearml/issues/539

[jkhenning]

Hi @Honzys,

In your case, you need the value of this environment variable to be be the path to the python executable of the venv you preinstalled in the docker image, something like CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=/home/user/venv/bin/python

[Honzys]

Hello,

sorry for the delayed reply. This actually helps!

But we've found out, that there's still the same problem if we abort, reset the experiment and we try to enqueue it again. It still tries to install some dependencies which breaks the whole venv.

Primarily we would like to skip any installation of any packages. But if that's not possible we would like to somehow control which dependencies will be installed.

There's a issue with typing package which is third party and for python>=3.5 it is not recommended to be installed - https://pypi.org/project/typing/ .

Shouldn't there be a condition in the requirements for the clearml-agent? Something like this:

typing;python_version<"3.5"

Which is recommended by the maintainer of the 3rd party of typing package?

Otherwise we get this error:

/opt/venv/bin/python /opt/venv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --no-warn-script-location --no-binary :none: --only-binary :none: -i https://nexus3.ba.innovatrics.net/repository/pypi-internal/simple --extra-index-url https://pypi.org/simple/ --extra-index-url https://nexus3.ba.innovatrics.net/repository/pypi-internal/simple --trusted-host nexus3.ba.innovatrics.net -- 'setuptools >= 42.0.0' wheel                                                                         
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/venv/lib/python3.8/site-packages/pip/__main__.py", line 29, in <module>
    from pip._internal.cli.main import main as _main
  File "/opt/venv/lib/python3.8/site-packages/pip/__init__.py", line 1, in <module>
    from typing import List, Optional
  File "/opt/venv/lib/python3.8/site-packages/typing.py", line 1359, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/opt/venv/lib/python3.8/site-packages/typing.py", line 1007, in __new__
    self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'

If you only run:

rm -rf /opt/venv/bin/lib/python3.8/site-packages/typing.py

It is fixed and it runs without any issues.

[lpogo]

Hi,

should CLEARML_AGENT_SKIP_PIP_VENV_INSTALL works also for conda? If not how can I handle this?

Thanks!

[lpogo]

Hi,

should CLEARML_AGENT_SKIP_PIP_VENV_INSTALL works also for conda? If not how can I handle this?

Thanks!

Answering my own question.. :) even if you are using conda environment on agent machine, set:

python_binary: "/usr/local/env/my_conda_env/bin/python3.8"
package_manager: {
    type: pip,
    system_site_packages: true,
    ....
}

Agent will use python binary from your conda environment with access to all libraries installed in conda env my_conda_env. Hope it helps sb.

[jkhenning]

Hi @Honzys,

Sorry, missed it somehow 🙁

But we've found out, that there's still the same problem if we abort, reset the experiment and we try to enqueue it again. It still tries to install some dependencies which breaks the whole venv.

Are you saying it behaves differently if the agent executes a newly enqueued experiment or a afiled experiment that was reset?

[odulcy-mindee]

@Honzys Hello :wave:

It would be great to have some option to disable all further installation / preparation of python evnrinoment and use the prepared env as it is. Is it possible now? Or are you thinking about adding this feature in som future version?

I got the same problem. I have a Docker image with everything inside and I needed to disable the pip install from the agent.

I saw this part in ClearML Agent: https://github.com/allegroai/clearml-agent/blob/1163c964389b8a5bd7205b473a8f550d88e401dd/clearml_agent/commands/worker.py#L2200 and https://github.com/allegroai/clearml-agent/blob/a51f9bed490124bdb290be972c3bcce6b90e7015/clearml_agent/definitions.py#L138

so by setting CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=1, I was able to disable the pip install. I also noticed that this environment variable is not in https://clear.ml/docs/latest/docs/configs/env_vars/ .

However, setting CLEARML_AGENT_SKIP_PIP_VENV_INSTALL didn't work for me, i.e. it used the binary that I provided to create a virtualenv but it didn't skip the installation. Also note that I used Task.execute_remotely from my code.

@jkhenning Could you confirm the purpose of CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL environment variable ?

[jkhenning]

Hi @odulcy-mindee,

I also noticed that this environment variable is not in https://clear.ml/docs/latest/docs/configs/env_vars/

We're working on adding this to the docs 🙂

Regarding the env var, CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL is designed to make the agent use the environment as is, without installing anything, so I think this is what you're after.

[odulcy-mindee]

Awesome, thank you @jkhenning !!

[KhanMechAI]

So this option has been removed in latest release? How else can we skip the installation