pip packages not working on long paths

[danielfrg]

The pip section in packages is not working correctly.

Affects: anaconda-project==0.8.0

Example project yaml:

name: 22-classification-and-regression-trees

commands:
  default:
    notebook: notebook.ipynb
    env_spec: default

env_specs:
  default:
    description: Default environment spec for running commands
    packages:
    - notebook
    - pandas
    - scikit-learn
    - statsmodels
    - matplotlib
    - seaborn
    - pydotplus
    - py-xgboost
    - graphviz
    - pip:
      - CHAID==4.0.0
      - word2vec
    channels: []
    platforms:
    - linux-64
    - osx-64
    - win-64

Doing prepare will pip install both CHAID and word2vec in the environment where anaconda-project is installed and not in the default environment.

[havocp]

seems likely to be somehow related to pip version

[danielfrg]

I am using pip 9.0.1. Maybe if I downgrade python to downgrade pip?

[danielfrg]

After looking at one project that does work with pip and worked for me in the same setup I tried a bunch of stuff the actual error was the directory name that has the anaconda-project.yml.

I had a 22-classification-and-regression-trees directory and if i change it to 22-classification it works fine.

I don't know if its the length of the whole path or just the last directory name or if its pip or what but I just tried this multiple times and I can reproduce (on MacOS).

[danielfrg]

I honestly don't know whats happening anymore :) but this combination of settings made it work for me:

name: 22-classification-and-regression

commands:
  default:
    notebook: classification-and-regression.ipynb
    env_spec: default

packages:
  - python=3.5
  - notebook
  - pandas
  - scipy
  - scikit-learn
  - statsmodels
  - matplotlib
  - seaborn
  - pydotplus
  - py-xgboost
  - graphviz
  - pip:
    - treelib
    - CHAID

env_specs:
  default:
    channels: []
    packages: []

[danielfrg]

Another example that doesn't work

name: 28-thrid-party

commands:
  default:
    unix: python main.py

packages:
  - python=3.5
  - notebook
  - pandas
  - pip:
    - dash

env_specs:
  default:
    channels: []
    packages: []

Now I actually don't know I thought it was the python version that made the previous one work but i guess not...

[danielfrg]

Ok, scratch the previous comment, that anaconda-project file works just fine, they all are probably valid and work fine.

I didn't want to believe it but it is indeed the length of the path to the environment that is created. I had a bunch of subdirectories to the path to that anaconda-project yaml and then I renamed one big directory to one character and it all worked fine.

I had no spaces and no weird characters in the path, i am also not sure what the number is where it breaks but I have seen it to many times to not believe it XD

[havocp]

There are operating system limits on path length and command line length; and conda has a limit on path length because it leaves placeholder paths of that length in binaries. Conda limit is the lowest, potentially.

Error handling/message at least should be fixed here.

[danielfrg]

I would have bet that I have done longer paths in the past, with java projects but i guess not. Is this a conda limit? because conda install of pkgs works just fine its pip that fails.

[havocp]

I'm not sure which limit is causing the problem...