conda skeleton pypi should respect setup_requires as a build-time requirement

[dturaev]

Expected behavior: conda skeleton pypi seqmagick produces a meta.yaml file that is used as input for conda build

Actual behavior: conda build fails because of missing dependency

Steps to reproduce: I followed the guide (https://conda.io/docs/user-guide/tutorials/build-pkgs-skeleton.html) to prepare a recipe for seqmagick (https://pypi.python.org/pypi/seqmagick/0.7.0). After the skeleton command generated meta.yaml, the build command failed with a somewhat cryptic error:

...
Source cache directory is: /apps/anaconda3/conda-bld/src_cache
Found source in cache: seqmagick-0.7.0_e5bd29a5bf.tar.gz
Extracting download
source tree in: /apps/anaconda3/conda-bld/seqmagick_1521544476937/work
Traceback (most recent call last):
  File "setup.py", line 47, in <module>
    license="GPL V3")
  File "/apps/anaconda3/conda-bld/seqmagick_1521544476937/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.6/site-packages/setuptools/__init__.py", line 128, in setup
    _install_setup_requires(attrs)
  File "/apps/anaconda3/conda-bld/seqmagick_1521544476937/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.6/site-packages/setuptools/__init__.py", line 123, in _install_setup_requires
    dist.fetch_build_eggs(dist.setup_requires)
  File "/apps/anaconda3/conda-bld/seqmagick_1521544476937/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.6/site-packages/setuptools/dist.py", line 461, in fetch_build_eggs
    replace_conflicting=True,
  File "/apps/anaconda3/conda-bld/seqmagick_1521544476937/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.6/site-packages/pkg_resources/__init__.py", line 866, in resolve
    replace_conflicting=replace_conflicting
  File "/apps/anaconda3/conda-bld/seqmagick_1521544476937/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1146, in best_match
    return self.obtain(req, installer)
  File "/apps/anaconda3/conda-bld/seqmagick_1521544476937/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1158, in obtain
    return installer(requirement)
  File "/apps/anaconda3/conda-bld/seqmagick_1521544476937/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.6/site-packages/setuptools/dist.py", line 528, in fetch_build_egg
    return cmd.easy_install(req)
  File "/apps/anaconda3/conda-bld/seqmagick_1521544476937/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.6/site-packages/setuptools/command/easy_install.py", line 661, in easy_install
    not self.always_copy, self.local_index
  File "/apps/anaconda3/conda-bld/seqmagick_1521544476937/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.6/site-packages/setuptools/package_index.py", line 660, in fetch_distribution
    self.find_packages(requirement)
  File "/apps/anaconda3/conda-bld/seqmagick_1521544476937/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.6/site-packages/setuptools/package_index.py", line 493, in find_packages
    self.scan_url(self.index_url + requirement.unsafe_name + '/')
  File "/apps/anaconda3/conda-bld/seqmagick_1521544476937/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.6/site-packages/setuptools/package_index.py", line 833, in scan_url
    self.process_url(url, True)
  File "/apps/anaconda3/conda-bld/seqmagick_1521544476937/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.6/site-packages/setuptools/package_index.py", line 328, in process_url
    "Be sure to add all dependencies in the meta.yaml  url=%s" % url)
RuntimeError: Setuptools downloading is disabled in conda build. Be sure to add all dependencies in the meta.yaml  url=https://pypi.python.org/simple/nose/
Traceback (most recent call last):
  File "/apps/anaconda3/bin/conda-build", line 11, in <module>
    sys.exit(main())
  File "/apps/anaconda3/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 420, in main
    execute(sys.argv[1:])
  File "/apps/anaconda3/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 411, in execute
    verify=args.verify)
  File "/apps/anaconda3/lib/python3.6/site-packages/conda_build/api.py", line 199, in build
    notest=notest, need_source_download=need_source_download, variants=variants)
  File "/apps/anaconda3/lib/python3.6/site-packages/conda_build/build.py", line 2097, in build_tree
    notest=notest,
  File "/apps/anaconda3/lib/python3.6/site-packages/conda_build/build.py", line 1370, in build
    utils.check_call_env(cmd, env=env, cwd=src_dir, stats=build_stats)
  File "/apps/anaconda3/lib/python3.6/site-packages/conda_build/utils.py", line 294, in check_call_env
    return _func_defaulting_env_to_os_environ('call', *popenargs, **kwargs)
  File "/apps/anaconda3/lib/python3.6/site-packages/conda_build/utils.py", line 273, in _func_defaulting_env_to_os_environ
    raise subprocess.CalledProcessError(proc.returncode, _args)
subprocess.CalledProcessError: Command '['/bin/bash', '-e', '/apps/anaconda3/conda-bld/seqmagick_1521544476937/work/conda_build.sh']' returned non-zero exit status 1.
...

After adding "nose" to the meta.yaml dependencies, conda build worked. (Also the Python version wasn't specified in meta.yaml, at least py3.4 is required; shouldn't it contain a line like

build:
      skip: True # [py27]

or how does conda otherwise know which Python version is required?) Conda info:

    active environment : test
    active env location : /apps/anaconda3/envs/test
            shell level : 1
       user config file : /root/.condarc
 populated config files : /apps/anaconda3/.condarc
                          /root/.condarc
          conda version : 4.4.11
    conda-build version : 3.7.1
         python version : 3.6.4.final.0
       base environment : /apps/anaconda3  (writable)
           channel URLs : https://repo.continuum.io/pkgs/main/linux-64
                          https://repo.continuum.io/pkgs/main/noarch
                          https://repo.continuum.io/pkgs/free/linux-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/r/linux-64
                          https://repo.continuum.io/pkgs/r/noarch
                          https://repo.continuum.io/pkgs/pro/linux-64
                          https://repo.continuum.io/pkgs/pro/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/bioconda/linux-64
                          https://conda.anaconda.org/bioconda/noarch
          package cache : /apps/anaconda3/pkgs
                          /root/.conda/pkgs
       envs directories : /apps/anaconda3/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/4.4.11 requests/2.18.4 CPython/3.6.4 Linux/4.15.6-1.el7.elrepo.x86_64 centos/7 glibc/2.17
                UID:GID : 0:0
             netrc file : None
           offline mode : False
```

[msarahan]

Skeletons are based on information provided in the package's setup.py. If that doesn't list nose as a dependency, conda-build will not add it. Imports in setup.py don't count, though perhaps with some finagling they could be taken into account. This is why the --extra-specs flag was added:

  --extra-specs EXTRA_SPECS
                        Extra specs for the build environment to extract the
                        skeleton.

It is not a bug that the skeleton does not automatically work. Skeletons are not guaranteed to work out-of-the-box. They are intended to give you a good starting place and save you some time. The problem space is too large to guarantee anything.

Adding the python version restriction would be nice, but again, there are many ways to specify this, and we can't promise to cover them all. PRs would be welcome to improve the python skeleton generator. If you'd like, we can change the title of this issue to be a feature request for that.

[dturaev]

I see, thanks a lot for the explanation! I just looked into the setup.py of seqmagick (https://github.com/fhcrc/seqmagick/blob/master/setup.py), it says

setup(name='seqmagick',
 ...
 setup_requires=['nose>=1.0'],
 python_requires='>=3.4',
 test_suite='nose.collector',
 install_requires=['biopython>=1.70'],
 ...
 license="GPL V3")

So the skeleton looks at the "install_requires" but not at the "setup_requires" attribute, is that right? Sure, feature request sounds like a good idea. Also it might be worth adding this explanation to https://conda.io/docs/user-guide/tutorials/build-pkgs-skeleton.html, this would help newbies like me. :)

[github-actions[bot]]

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

1. Verify that you can still reproduce the issue at hand
2. Comment that the issue is still reproducible and include: - What OS and version you reproduced the issue on - What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!