conda env create -f environment.yml fails during pip step

[jcwomack]

MacOS 10.14 Mojave Miniconda3 (conda 4.8.3)

Following the installation instructions, conda env create -f environment.yml fails during the installation of the tokenizers package:

error: Can not find Rust compiler
  ----------------------------------------
  ERROR: Failed building wheel for tokenizers
ERROR: Could not build wheels for tokenizers which use PEP 517 and cannot be installed directly


CondaEnvException: Pip failed

It seems likely that this is related to the an issue reported for transformers (https://github.com/huggingface/transformers/issues/2831).

Installation of a Rust compiler may be required to proceed. Possibly this can be done through conda?

[jcwomack]

I was able to create the environment by adding rust 1.46.0 as a conda dependency. I also found I needed to update the version of transformers in the list of pip dependencies to avoid an issue with PyO3 requiring a development version of the Rust compiler.

diff --git a/environment.yml b/environment.yml
index ed38bc8..7ed7fe2 100644
--- a/environment.yml
+++ b/environment.yml
@@ -32,6 +32,7 @@ dependencies:
   - tqdm=4.43
   - wheel
   - widgetsnbextension=3.5
+  - rust=1.46.0
   - pip:
     - atari-py==0.2.6
     - ftfy==5.7
@@ -53,5 +54,5 @@ dependencies:
     - tensorflow==2.1.0 # or tensorflow-gpu if gpu
     - tf-agents==0.3.0
     - tfx==0.21.2
-    - transformers==2.8.0
+    - transformers==3.2.0
     - urlextract==0.14.0

I have not tested all the notebooks in the environment, so it is possible that these changes cause something to break.

[RichardShea]

Yes, I had the same issue (Mohave with Anaconda) and your conclusion on rust and transformers worked for me so far.

Attached is the environment.yml file (I had to zip it since it doesn't support uploading yml) that I added rust=1.46.0 and changed transformers to 3.2.0.

Thank you!

environment.yml.zip

[ageron]

Thanks for your feedback. I updated environment.yml to recent versions of all libraries, and I ran some tests. Everything seems to be working fine, and in particular the Transformers example (at the end of chapter 16) works great. Hope this helps.