deep-symbolic-optimization
deep-symbolic-optimization copied to clipboard
brendenpetersen
Trouble Installing, sample virtual environment config?
Hello.
I have tried with Python version 3.11, 3.9, 3.8, and now 3.7, with pip versions 23.2.1 and 20.3.4, and wheel versions 0.36.2 and 0.41.1, and numpy versions 1.16 - 1.19 and for the life of me no matter how many different clean-slate virtual environments I use I can't seem to get pip install -e . to work. For example, with pip-23.2.1 I get:
Obtaining file:///Users/edwardfinkelstein/deep-symbolic-optimization/dso
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
Getting requirements to build editable ... error
error: subprocess-exited-with-error
× Getting requirements to build editable did not run successfully.
│ exit code: 1
╰─> [53 lines of output]
/Users/edwardfinkelstein/deepSRvenv/bin/python3.7: No module named pip
/Users/edwardfinkelstein/deep-symbolic-optimization/dso/setup.py:5: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
!!
********************************************************************************
Requirements should be satisfied by a PEP 517 installer.
If you are using pip, you can try `pip install --use-pep517`.
********************************************************************************
!!
dist.Distribution().fetch_build_eggs(['Cython', 'numpy'])
Traceback (most recent call last):
File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-184jyeef/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 96, in _fetch_build_egg_no_warn
subprocess.check_call(cmd)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 328, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/Users/edwardfinkelstein/deepSRvenv/bin/python3.7', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/tmp7sr8fk9_', '--quiet', 'numpy']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/edwardfinkelstein/deepSRvenv/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
main()
File "/Users/edwardfinkelstein/deepSRvenv/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/Users/edwardfinkelstein/deepSRvenv/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 132, in get_requires_for_build_editable
return hook(config_settings)
File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-184jyeef/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 450, in get_requires_for_build_editable
return self.get_requires_for_build_wheel(config_settings)
File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-184jyeef/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=['wheel'])
File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-184jyeef/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
self.run_setup()
File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-184jyeef/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 488, in run_setup
self).run_setup(setup_script=setup_script)
File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-184jyeef/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 338, in run_setup
exec(code, locals())
File "<string>", line 5, in <module>
File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-184jyeef/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 907, in fetch_build_eggs
return _fetch_build_eggs(self, requires)
File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-184jyeef/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 41, in _fetch_build_eggs
replace_conflicting=True,
File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-184jyeef/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 830, in resolve
req, best, replace_conflicting, env, installer, required_by, to_activate
File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-184jyeef/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 866, in _resolve_dist
req, ws, installer, replace_conflicting=replace_conflicting
File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-184jyeef/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1135, in best_match
return self.obtain(req, installer)
File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-184jyeef/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1147, in obtain
return installer(requirement)
File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-184jyeef/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 98, in _fetch_build_egg_no_warn
raise DistutilsError(str(e)) from e
distutils.errors.DistutilsError: Command '['/Users/edwardfinkelstein/deepSRvenv/bin/python3.7', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/tmp7sr8fk9_', '--quiet', 'numpy']' returned non-zero exit status 1.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build editable did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Doing pip install --use-pep517 -e . changes nothing. With pip version 20.3.4 I get:
Installing build dependencies ... done
Getting requirements to build wheel ... error
ERROR: Command errored out with exit status 1:
command: /Users/edwardfinkelstein/deepSRvenv/bin/python3.7 /Users/edwardfinkelstein/deepSRvenv/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/tmpdfdn1xir
cwd: /Users/edwardfinkelstein/deep-symbolic-optimization/dso
Complete output (51 lines):
/Users/edwardfinkelstein/deepSRvenv/bin/python3.7: No module named pip
/Users/edwardfinkelstein/deep-symbolic-optimization/dso/setup.py:5: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
!!
********************************************************************************
Requirements should be satisfied by a PEP 517 installer.
If you are using pip, you can try `pip install --use-pep517`.
********************************************************************************
!!
dist.Distribution().fetch_build_eggs(['Cython', 'numpy'])
Traceback (most recent call last):
File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-8nqb_t8o/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 96, in _fetch_build_egg_no_warn
subprocess.check_call(cmd)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 328, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/Users/edwardfinkelstein/deepSRvenv/bin/python3.7', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/tmpu1788i4d', '--quiet', 'numpy']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/edwardfinkelstein/deepSRvenv/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
main()
File "/Users/edwardfinkelstein/deepSRvenv/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/Users/edwardfinkelstein/deepSRvenv/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 114, in get_requires_for_build_wheel
return hook(config_settings)
File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-8nqb_t8o/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=['wheel'])
File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-8nqb_t8o/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
self.run_setup()
File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-8nqb_t8o/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 488, in run_setup
self).run_setup(setup_script=setup_script)
File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-8nqb_t8o/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 338, in run_setup
exec(code, locals())
File "<string>", line 5, in <module>
File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-8nqb_t8o/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 907, in fetch_build_eggs
return _fetch_build_eggs(self, requires)
File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-8nqb_t8o/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 41, in _fetch_build_eggs
replace_conflicting=True,
File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-8nqb_t8o/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 830, in resolve
req, best, replace_conflicting, env, installer, required_by, to_activate
File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-8nqb_t8o/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 866, in _resolve_dist
req, ws, installer, replace_conflicting=replace_conflicting
File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-8nqb_t8o/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1135, in best_match
return self.obtain(req, installer)
File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-8nqb_t8o/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1147, in obtain
return installer(requirement)
File "/private/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/pip-build-env-8nqb_t8o/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 98, in _fetch_build_egg_no_warn
raise DistutilsError(str(e)) from e
distutils.errors.DistutilsError: Command '['/Users/edwardfinkelstein/deepSRvenv/bin/python3.7', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/tmpu1788i4d', '--quiet', 'numpy']' returned non-zero exit status 1.
----------------------------------------
WARNING: Discarding file:///Users/edwardfinkelstein/deep-symbolic-optimization/dso. Command errored out with exit status 1: /Users/edwardfinkelstein/deepSRvenv/bin/python3.7 /Users/edwardfinkelstein/deepSRvenv/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/tmpdfdn1xir Check the logs for full command output.
ERROR: Command errored out with exit status 1: /Users/edwardfinkelstein/deepSRvenv/bin/python3.7 /Users/edwardfinkelstein/deepSRvenv/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/n1/0_k13bw50977fy1vswx1ln4m0000gn/T/tmpdfdn1xir Check the logs for full command output.
WARNING: You are using pip version 20.3.4; however, version 23.2.1 is available.
You should consider upgrading via the '/Users/edwardfinkelstein/deepSRvenv/bin/python3.7 -m pip install --upgrade pip' command.
I've followed the steps in the README every time to no avail. Computer info for my Apple M1 (uname -a):
Darwin Edwards-MBP.webspeed.dk 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun 8 22:22:19 PDT 2023; root:xnu-8796.121.3~7/RELEASE_ARM64_T8103 arm64
Does anyone by chance have a python/pip/wheel/numpy/anything_else_relevant version combination that worked for them? Any help's greatly appreciated and would be happy to answer any questions that might help further diagnose the issue.
I had the same error following the README instructions. What worked for me was doing
/usr/bin/python3.6 -m venv venv-dso-3.6
(or whereever you have python3.6 located)
instead of just python. I think some of the packages depend on version 3.6. Then when actually running dso inside the venv, I also specify python3.6.
@edfink234, did you eventually manage? I'm having the same issues, and running on python3.6 as @mtrautner suggested also does not work for me.
Kind Regards, Jeroen
I also had the same issue and gave up on linux (as my linux distro requires 3.10 and i dont wanna destroy it) because of the requirements and dependencys... so i created a docker file to run it in a container. Maybe this might help you too...
FROM ubuntu:18.04
RUN apt update
RUN apt install python3-pip software-properties-common -y
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
RUN ln -s /usr/bin/pip3 /usr/bin/pip
RUN ln -s /usr/bin/python3.6 /usr/bin/python
RUN pip install --upgrade setuptools pip
WORKDIR /code
COPY ./deep-symbolic-optimization/dso ./dso
RUN pip install -e ./dso
# COPY ./mydata ./mydata
# python -m dso.run /code/mydata/Data1.json
CMD ["python3", "--version"]
after that you can either edit the dso and rebuild the container, or link the mydata folder to /code like i do....
docker run -it --name ia-dso -v ./mydata:/code/mydata dso:latest /bin/bash
after that you are in the interactive shell and can use
python -m dso.run /code/mydata2/Data1.json --runs 100
to make your first run...
@edfink234 What OS are you using?
@snapo Thank you very much for providing a Dockerfile. I think we could take that as a PR if you'd like to create one.
Similar for any improvements to the README! Internally, we have a slightly different setup, so it can be hard for us to keep the public README up to date.
@brendenpetersen i was thinking about creating a PR... but i have to do some more work on the docker file to make it much more easy to scale / run tasks in the future and make it able to orchestrate tasks with different seeds via docker compose. When its ready yes i will do it, this is just a quick and dirty hackup to test the framework. with docker compose/kubernetis you would then be able to distribute each search to the cloud where each seed runs in its own docker container. (at least what i am planning).
This would allow you then to have datafile , json config
then lets say run it on 1k amazon cloud instances where each instance has its own seed for 1 run. (therefore i have to still think about a little to collect data after the run , maybe i am script something ....). With that setup you would then be able to say 30 runs per instance, 10k instances and have a huge time reduction for the serach space.
I got a similar error with python 3.9 on both windows and ubuntu os. The error disappeared if I installed it with python 3.6.
Error info:
Obtaining file:///home/gpzhao/Github/deep-symbolic-optimization-master/dso Preparing metadata (setup.py) ... done Collecting pytest (from dso==1.0.dev0) Using cached pytest-7.4.3-py3-none-any.whl.metadata (7.9 kB) Requirement already satisfied: cython in /home/gpzhao/.conda/envs/dsr39/lib/python3.9/site-packages (from dso==1.0.dev0) (3.0.7) Collecting numpy<=1.19 (from dso==1.0.dev0) Using cached numpy-1.19.0.zip (7.3 MB) Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... error error: subprocess-exited-with-error
× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [54 lines of output]
Running from numpy source directory.
Error compiling Cython file:
------------------------------------------------------------
...
self.rng_state.ctr.v[i] = counter[i]
self._reset_state_variables()
self._bitgen.state = <void *>&self.rng_state
self._bitgen.next_uint64 = &philox_uint64
^
------------------------------------------------------------
_philox.pyx:195:35: Cannot assign type 'uint64_t (*)(void *) except? -1 nogil' to 'uint64_t (*)(void *) noexcept nogil'. Exception values are incompatible. Suggest adding 'noexcept' to type 'uint64_t (void *) except? -1 nogil'.
Processing numpy/random/_bounded_integers.pxd.in
Processing numpy/random/_philox.pyx
Traceback (most recent call last):
File "/tmp/pip-install-s5u3gw4k/numpy_c70991b77c1648a99494c52834ba182a/tools/cythonize.py", line 235, in <module>
main()
File "/tmp/pip-install-s5u3gw4k/numpy_c70991b77c1648a99494c52834ba182a/tools/cythonize.py", line 231, in main
find_process_files(root_dir)
File "/tmp/pip-install-s5u3gw4k/numpy_c70991b77c1648a99494c52834ba182a/tools/cythonize.py", line 222, in find_process_files
process(root_dir, fromfile, tofile, function, hash_db)
File "/tmp/pip-install-s5u3gw4k/numpy_c70991b77c1648a99494c52834ba182a/tools/cythonize.py", line 188, in process
processor_function(fromfile, tofile)
File "/tmp/pip-install-s5u3gw4k/numpy_c70991b77c1648a99494c52834ba182a/tools/cythonize.py", line 77, in process_pyx
subprocess.check_call(
File "/home/gpzhao/.conda/envs/dsr39/lib/python3.9/subprocess.py", line 373, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/gpzhao/.conda/envs/dsr39/bin/python', '-m', 'cython', '-3', '--fast-fail', '-o', '_philox.c', '_philox.pyx']' returned non-zero exit status 1.
Cythonizing sources
Traceback (most recent call last):
File "/home/gpzhao/.conda/envs/dsr39/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
main()
File "/home/gpzhao/.conda/envs/dsr39/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/home/gpzhao/.conda/envs/dsr39/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
return hook(metadata_directory, config_settings)
File "/tmp/pip-build-env-4tnhlnaz/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 366, in prepare_metadata_for_build_wheel
self.run_setup()
File "/tmp/pip-build-env-4tnhlnaz/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 480, in run_setup
super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
File "/tmp/pip-build-env-4tnhlnaz/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 311, in run_setup
exec(code, locals())
File "<string>", line 489, in <module>
File "<string>", line 469, in setup_package
File "<string>", line 274, in generate_cython
RuntimeError: Running cythonize failed!
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed
× Encountered error while generating package metadata. ╰─> See above for output.
note: This is an issue with the package mentioned above, not pip. hint: See above for details.
