Installation problems

[Dbg-atlas]

I've been trying to install Keras and Tensorflow from R following the instructions in the book, but I'm encountering an issue that I can't solve even following the discussion here.

First I installed the libraries on Windows using RStudio and everything worked perfectly. However, I only got it to work on CPU, not GPU. To facilitate the installation of dependencies to use the GPU I followed the instructions in the book and used Windows Subsystem for Linux to install Linux on my computer.

Once I did this, using Visual Studio Code with WSL to run the same code, the installation does not work. The end result is as follows:

Error: Valid installation of TensorFlow not found.

Python environments searched for 'tensorflow' package:
 /home/dberm/.pyenv/versions/3.9.16/bin/python3.9
 /usr/bin/python3.10

My full code so far:

install.packages("keras")

Installing package into ‘/home/dberm/R/x86_64-pc-linux-gnu-library/4.1’ (as ‘lib’ is unspecified) trying URL 'https://cloud.r-project.org/src/contrib/keras_2.11.1.tar.gz' Content type 'application/x-gzip' length 3527604 bytes (3.4 MB)

downloaded 3.4 MB

• installing source package ‘keras’ ... ** package ‘keras’ successfully unpacked and MD5 sums checked ** using staged installation ** R ** inst ** byte-compile and prepare package for lazy loading ** help *** installing help indices ** building package indices ** installing vignettes ** testing if installed package can be loaded from temporary location ** testing if installed package can be loaded from final location ** testing if installed package keeps a record of temporary installation path
• DONE (keras)

The downloaded source packages are in ‘/tmp/RtmpDj8V9u/downloaded_packages’

library(reticulate) virtualenv_create("r-reticulate", python = install_python())

virtualenv: r-reticulate

library(keras)
install_keras(envname = "r-reticulate")

Using virtual environment 'r-reticulate' ...

• '/home/dberm/.virtualenvs/r-reticulate/bin/python' -m pip install --upgrade --no-user --ignore-installed 'tensorflow==2.11.' 'tensorflow-hub' 'tensorflow-datasets' 'scipy' 'requests' 'Pillow' 'h5py' 'pandas' 'pydot' Collecting tensorflow==2.11. Using cached tensorflow-2.11.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (588.3 MB) Collecting tensorflow-hub Using cached tensorflow_hub-0.13.0-py2.py3-none-any.whl (100 kB) Collecting tensorflow-datasets Using cached tensorflow_datasets-4.8.3-py3-none-any.whl (5.4 MB) Collecting scipy Using cached scipy-1.10.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB) Collecting requests Using cached requests-2.28.2-py3-none-any.whl (62 kB) Collecting Pillow Using cached Pillow-9.4.0-cp39-cp39-manylinux_2_28_x86_64.whl (3.4 MB) Collecting h5py Using cached h5py-3.8.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.7 MB) Collecting pandas Using cached pandas-1.5.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.2 MB) Collecting pydot Using cached pydot-1.4.2-py2.py3-none-any.whl (21 kB) Collecting astunparse>=1.6.0 Using cached astunparse-1.6.3-py2.py3-none-any.whl (12 kB) Collecting opt-einsum>=2.3.2 Using cached opt_einsum-3.3.0-py3-none-any.whl (65 kB) Collecting typing-extensions>=3.6.6 Using cached typing_extensions-4.5.0-py3-none-any.whl (27 kB) Collecting google-pasta>=0.1.1 Using cached google_pasta-0.2.0-py3-none-any.whl (57 kB) Collecting grpcio<2.0,>=1.24.3 Using cached grpcio-1.51.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.8 MB) Collecting tensorflow-estimator<2.12,>=2.11.0 Using cached tensorflow_estimator-2.11.0-py2.py3-none-any.whl (439 kB) Collecting termcolor>=1.1.0 Using cached termcolor-2.2.0-py3-none-any.whl (6.6 kB) Collecting libclang>=13.0.0 Using cached libclang-15.0.6.1-py2.py3-none-manylinux2010_x86_64.whl (21.5 MB) Collecting packaging Using cached packaging-23.0-py3-none-any.whl (42 kB) Collecting tensorboard<2.12,>=2.11 Using cached tensorboard-2.11.2-py3-none-any.whl (6.0 MB) Collecting wrapt>=1.11.0 Using cached wrapt-1.15.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (78 kB) Collecting flatbuffers>=2.0 Using cached flatbuffers-23.3.3-py2.py3-none-any.whl (26 kB) Collecting tensorflow-io-gcs-filesystem>=0.23.1 Using cached tensorflow_io_gcs_filesystem-0.31.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.4 MB) Collecting numpy>=1.20 Using cached numpy-1.24.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB) Collecting protobuf<3.20,>=3.9.2 Using cached protobuf-3.19.6-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB) Collecting absl-py>=1.0.0 Using cached absl_py-1.4.0-py3-none-any.whl (126 kB) Collecting keras<2.12,>=2.11.0 Using cached keras-2.11.0-py2.py3-none-any.whl (1.7 MB) Collecting setuptools Using cached setuptools-67.6.0-py3-none-any.whl (1.1 MB) Collecting gast<=0.4.0,>=0.2.1 Using cached gast-0.4.0-py3-none-any.whl (9.8 kB) Collecting six>=1.12.0 Using cached six-1.16.0-py2.py3-none-any.whl (11 kB) Collecting promise Using cached promise-2.3.tar.gz (19 kB) Preparing metadata (setup.py) ... done Collecting tqdm Using cached tqdm-4.65.0-py3-none-any.whl (77 kB) Collecting dm-tree Using cached dm_tree-0.1.8-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (153 kB) Collecting tensorflow-metadata Using cached tensorflow_metadata-1.12.0-py3-none-any.whl (52 kB) Collecting toml Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB) Collecting etils[enp,epath]>=0.9.0 Using cached etils-1.1.1-py3-none-any.whl (115 kB) Collecting psutil Using cached psutil-5.9.4-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (280 kB) Collecting click Using cached click-8.1.3-py3-none-any.whl (96 kB) Collecting charset-normalizer<4,>=2 Using cached charset_normalizer-3.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (199 kB) Collecting idna<4,>=2.5 Using cached idna-3.4-py3-none-any.whl (61 kB) Collecting certifi>=2017.4.17 Using cached certifi-2022.12.7-py3-none-any.whl (155 kB) Collecting urllib3<1.27,>=1.21.1 Using cached urllib3-1.26.15-py2.py3-none-any.whl (140 kB) Collecting pytz>=2020.1 Using cached pytz-2022.7.1-py2.py3-none-any.whl (499 kB) Collecting python-dateutil>=2.8.1 Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) Collecting pyparsing>=2.1.4 Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB) Collecting wheel<1.0,>=0.23.0 Using cached wheel-0.40.0-py3-none-any.whl (64 kB) Collecting zipp Using cached zipp-3.15.0-py3-none-any.whl (6.8 kB) Collecting importlib_resources Using cached importlib_resources-5.12.0-py3-none-any.whl (36 kB) Collecting tensorboard-data-server<0.7.0,>=0.6.0 Using cached tensorboard_data_server-0.6.1-py3-none-manylinux2010_x86_64.whl (4.9 MB) Collecting markdown>=2.6.8 Using cached Markdown-3.4.2-py3-none-any.whl (93 kB) Collecting tensorboard-plugin-wit>=1.6.0 Using cached tensorboard_plugin_wit-1.8.1-py3-none-any.whl (781 kB) Collecting google-auth<3,>=1.6.3 Using cached google_auth-2.16.2-py2.py3-none-any.whl (177 kB) Collecting google-auth-oauthlib<0.5,>=0.4.1 Using cached google_auth_oauthlib-0.4.6-py2.py3-none-any.whl (18 kB) Collecting werkzeug>=1.0.1 Using cached Werkzeug-2.2.3-py3-none-any.whl (233 kB) Collecting googleapis-common-protos<2,>=1.52.0 Using cached googleapis_common_protos-1.59.0-py2.py3-none-any.whl (223 kB) Collecting rsa<5,>=3.1.4 Using cached rsa-4.9-py3-none-any.whl (34 kB) Collecting pyasn1-modules>=0.2.1 Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB) Collecting cachetools<6.0,>=2.0.0 Using cached cachetools-5.3.0-py3-none-any.whl (9.3 kB) Collecting requests-oauthlib>=0.7.0 Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB) Collecting importlib-metadata>=4.4 Using cached importlib_metadata-6.1.0-py3-none-any.whl (21 kB) Collecting MarkupSafe>=2.1.1 Using cached MarkupSafe-2.1.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB) Collecting pyasn1<0.5.0,>=0.4.6 Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB) Collecting oauthlib>=3.0.0 Using cached oauthlib-3.2.2-py3-none-any.whl (151 kB) Building wheels for collected packages: promise Building wheel for promise (setup.py) ... error error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully. │ exit code: 1 ╰─> [34 lines of output] Traceback (most recent call last): File "", line 2, in File "", line 34, in File "/tmp/pip-install-c9bisgzm/promise_af6168e0cb2e42adaeca9a8025bee6aa/setup.py", line 28, in setup( File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/init.py", line 108, in setup return distutils.core.setup(**attrs) File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 172, in setup ok = dist.parse_command_line() File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 475, in parse_command_line args = self._parse_command_opts(parser, args) File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/dist.py", line 1119, in _parse_command_opts nargs = _Distribution._parse_command_opts(self, parser, args) File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 534, in _parse_command_opts cmd_class = self.get_command_class(command) File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/dist.py", line 966, in get_command_class self.cmdclass[command] = cmdclass = ep.load() File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/init.py", line 208, in load module = import_module(match.group('module')) File "/home/dberm/.pyenv/versions/3.9.16/lib/python3.9/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1030, in _gcd_import File "", line 1007, in _find_and_load File "", line 986, in _find_and_load_unlocked File "", line 680, in _load_unlocked File "", line 850, in exec_module File "", line 228, in _call_with_frames_removed File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 28, in from .macosx_libfile import calculate_macosx_platform_tag File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/wheel/macosx_libfile.py", line 43, in import ctypes File "/home/dberm/.pyenv/versions/3.9.16/lib/python3.9/ctypes/init.py", line 8, in from _ctypes import Union, Structure, Array ModuleNotFoundError: No module named '_ctypes' [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for promise Running setup.py clean for promise Failed to build promise Installing collected packages: tensorboard-plugin-wit, pytz, pyasn1, libclang, flatbuffers, dm-tree, zipp, wrapt, wheel, urllib3, typing-extensions, tqdm, toml, termcolor, tensorflow-io-gcs-filesystem, tensorflow-estimator, tensorboard-data-server, six, setuptools, rsa, pyparsing, pyasn1-modules, psutil, protobuf, Pillow, packaging, oauthlib, numpy, MarkupSafe, keras, idna, grpcio, gast, etils, click, charset-normalizer, certifi, cachetools, absl-py, werkzeug, tensorflow-hub, scipy, requests, python-dateutil, pydot, promise, opt-einsum, importlib_resources, importlib-metadata, h5py, googleapis-common-protos, google-pasta, google-auth, astunparse, tensorflow-metadata, requests-oauthlib, pandas, markdown, tensorflow-datasets, google-auth-oauthlib, tensorboard, tensorflow Running setup.py install for promise ... done DEPRECATION: promise was installed using the legacy 'setup.py install' method, because a wheel could not be built for it. pip 23.1 will enforce this behaviour change. A possible replacement is to fix the wheel build issue reported above. Discussion can be found at https://github.com/pypa/pip/issues/8368 Successfully installed MarkupSafe-2.1.2 Pillow-9.4.0 absl-py-1.4.0 astunparse-1.6.3 cachetools-5.3.0 certifi-2022.12.7 charset-normalizer-3.1.0 click-8.1.3 dm-tree-0.1.8 etils-1.1.1 flatbuffers-23.3.3 gast-0.4.0 google-auth-2.16.2 google-auth-oauthlib-0.4.6 google-pasta-0.2.0 googleapis-common-protos-1.59.0 grpcio-1.51.3 h5py-3.8.0 idna-3.4 importlib-metadata-6.1.0 importlib_resources-5.12.0 keras-2.11.0 libclang-15.0.6.1 markdown-3.4.2 numpy-1.24.2 oauthlib-3.2.2 opt-einsum-3.3.0 packaging-23.0 pandas-1.5.3 promise-2.3 protobuf-3.19.6 psutil-5.9.4 pyasn1-0.4.8 pyasn1-modules-0.2.8 pydot-1.4.2 pyparsing-3.0.9 python-dateutil-2.8.2 pytz-2022.7.1 requests-2.28.2 requests-oauthlib-1.3.1 rsa-4.9 scipy-1.10.1 setuptools-67.6.0 six-1.16.0 tensorboard-2.11.2 tensorboard-data-server-0.6.1 tensorboard-plugin-wit-1.8.1 tensorflow-2.11.1 tensorflow-datasets-4.8.3 tensorflow-estimator-2.11.0 tensorflow-hub-0.13.0 tensorflow-io-gcs-filesystem-0.31.0 tensorflow-metadata-1.12.0 termcolor-2.2.0 toml-0.10.2 tqdm-4.65.0 typing-extensions-4.5.0 urllib3-1.26.15 werkzeug-2.2.3 wheel-0.40.0 wrapt-1.15.0 zipp-3.15.0

Installation complete.

tensorflow::tf_config()

Valid installation of TensorFlow not found.

Python environments searched for 'tensorflow' package: /home/dberm/.pyenv/versions/3.9.16/bin/python3.9 /usr/bin/python3.10

Python exception encountered: Traceback (most recent call last): File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook return _run_hook(name, _hook) File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 93, in _run_hook module = hook() File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 117, in _hook return find_and_load(name, import) File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/tensorflow/init.py", line 37, in from tensorflow.python.tools import module_util as _module_util File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook return _run_hook(name, _hook) File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 93, in _run_hook module = hook() File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 117, in _hook return find_and_load(name, import) File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook return _run_hook(name, _hook) File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 93, in _run_hook module = hook() File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 117, in _hook return find_and_load(name, import) File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/tensorflow/python/init.py", line 24, in import ctypes File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook return _run_hook(name, _hook) File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 93, in _run_hook module = hook() File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 117, in _hook return find_and_load(name, import) File "/home/dberm/.pyenv/versions/3.9.16/lib/python3.9/ctypes/init.py", line 8, in from _ctypes import Union, Structure, Array File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook return _run_hook(name, _hook) File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 93, in _run_hook module = hook() File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 117, in _hook return find_and_load(name, import) ModuleNotFoundError: No module named '_ctypes'

You can install TensorFlow using the install_tensorflow() function.

If I try to use tensorflow::install_tensorflow() before tensorflow::tf_config() the result is the same. I also tried the instructions from the RStudio website, with the same results:

install.packages("tensorflow")

library(reticulate)
path_to_python <- install_python()
virtualenv_create("r-reticulate", python = path_to_python)

library(tensorflow)
install_tensorflow(envname = "r-reticulate")

install.packages("keras")
library(keras)
install_keras(envname = "r-reticulate")

tensorflow::tf_config()

As far as I can tell by the output, tensorflow::tf_config() is looking for Tensorflow here:

Python environments searched for 'tensorflow' package: /home/dberm/.pyenv/versions/3.9.16/bin/python3.9 /usr/bin/python3.10

while Tensorflor should be in envname = "r-reticulate", but I'm not used to work with Python enviroments so I don't know to change this behaviour.

I also read here this comment:

Additionally, you must be running an Arm native build of R, not the x86 build running under Rosetta.

Which I don't know if applies to my case or is just for Intel Mac.

I don't know what else to try as I have been several days stuck here and I lack the required Python knowledge to make any progress. Any help would be appreciated.