janclemenslab
DAS without the anaconda channel
Many institutions are blocking the anaconda channel because of licensing issues, so we need builds that work without anaconda. This is mainly an issue for windows because we rely on anaconda to provide the tensorflow-gpu package.
The new release 0.38.4 was built without anaconda in all supported platforms (linux, macos (arm only), windows).
You can give it a try with:
conda create -n das python=3.10 das=0.32.4 -c conda-forge -c ncb
Linux and Mac builds should also work with python=3.11.
Please test and report any issues with the no-anaconda builds here.
May also address #77.
Just commenting that this worked very well for me. Just a note: don't be surprised if the install takes much longer than previous installations of das...
Thanks! Glad it works.
And thanks for the note on the slowness: It didn't take longer on my machine. What version of conda are you on? The newer versions (23.10+) use the faster mamba solver by default (https://conda.org/blog/2023-11-06-conda-23-10-0-release/).
So either update conda (conda update conda -n base) or use mamba instead of conda for the installation. We will update the docs with this information once the new builds have been tested properly.
I am getting errors that the ncb channel does not include files for osx-arm64
conda install ncb::das
Hi, please use the installation instruction from the github readme or the documentation. This is the correct command:
conda create python=3.11 das=0.32.4 -c conda-forge -c ncb -c nvidia -c apple -n das -y
Just tested it on my M1 mac and it works. Please re-open if you run into issues with this.
Our institute blocks the default conda repo.
Collecting package metadata (current_repodata.json): failed
# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
Traceback (most recent call last):
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/gateways/repodata/__init__.py", line 161, in conda_http_errors
yield
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/gateways/repodata/__init__.py", line 127, in repodata
response.raise_for_status()
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://conda.anaconda.org/ncb/osx-arm64/current_repodata.json
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/requests/models.py", line 974, in json
return complexjson.loads(self.text, **kwargs)
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/simplejson/__init__.py", line 525, in loads
return _default_decoder.decode(s)
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/simplejson/decoder.py", line 372, in decode
obj, end = self.raw_decode(s)
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/simplejson/decoder.py", line 402, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/exceptions.py", line 1132, in __call__
return func(*args, **kwargs)
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/cli/main.py", line 69, in main_subshell
exit_code = do_call(args, p)
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/cli/conda_argparse.py", line 122, in do_call
return getattr(module, func_name)(args, parser)
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/notices/core.py", line 121, in wrapper
return func(*args, **kwargs)
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/cli/main_create.py", line 41, in execute
install(args, parser, 'create')
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/cli/install.py", line 264, in install
unlink_link_transaction = solver.solve_for_transaction(
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/core/solve.py", line 134, in solve_for_transaction
unlink_precs, link_precs = self.solve_for_diff(update_modifier, deps_modifier,
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/core/solve.py", line 177, in solve_for_diff
final_precs = self.solve_final_state(update_modifier, deps_modifier, prune, ignore_pinned,
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/core/solve.py", line 282, in solve_final_state
ssc = self._collect_all_metadata(ssc)
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/common/io.py", line 84, in decorated
return f(*args, **kwds)
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/core/solve.py", line 449, in _collect_all_metadata
index, r = self._prepare(prepared_specs)
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/core/solve.py", line 1064, in _prepare
reduced_index = get_reduced_index(self.prefix, self.channels,
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/core/index.py", line 267, in get_reduced_index
new_records = SubdirData.query_all(spec, channels=channels, subdirs=subdirs,
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/core/subdir_data.py", line 169, in query_all
result = tuple(chain.from_iterable(executor.map(subdir_query, channel_urls)))
File "/Users/abdelhaym/miniconda/lib/python3.9/concurrent/futures/_base.py", line 609, in result_iterator
yield fs.pop().result()
File "/Users/abdelhaym/miniconda/lib/python3.9/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/Users/abdelhaym/miniconda/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
File "/Users/abdelhaym/miniconda/lib/python3.9/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/core/subdir_data.py", line 158, in subdir_query
return tuple(
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/core/subdir_data.py", line 174, in query
self.load()
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/core/subdir_data.py", line 264, in load
_internal_state = self._load()
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/core/subdir_data.py", line 366, in _load
raw_repodata_str = self._repo.repodata(cache.state) # type: ignore
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/gateways/repodata/__init__.py", line 127, in repodata
response.raise_for_status()
File "/Users/abdelhaym/miniconda/lib/python3.9/contextlib.py", line 137, in __exit__
self.gen.throw(typ, value, traceback)
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/gateways/repodata/__init__.py", line 208, in conda_http_errors
raise RepodataIsEmpty(
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/conda/exceptions.py", line 462, in __init__
body = response.json()
File "/Users/abdelhaym/miniconda/lib/python3.9/site-packages/requests/models.py", line 978, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
`$ /Users/abdelhaym/miniconda/bin/conda create python=3.11 das=0.32.4 -c conda-forge -c ncb -c nvidia -c apple -n das -y`
environment variables:
CIO_TEST=<not set>
CONDA_DEFAULT_ENV=base
CONDA_EXE=/Users/abdelhaym/miniconda/bin/conda
CONDA_PREFIX=/Users/abdelhaym/miniconda
CONDA_PROMPT_MODIFIER=(base)
CONDA_PYTHON_EXE=/Users/abdelhaym/miniconda/bin/python
CONDA_ROOT=/Users/abdelhaym/miniconda
CONDA_SHLVL=1
CURL_CA_BUNDLE=<not set>
LD_PRELOAD=<not set>
PATH=/Users/abdelhaym/bin:/Users/abdelhaym/homebrew/bin:/Users/abdelhaym/.c
argo/bin:/Users/abdelhaym/homebrew/opt/qt@5/bin:/Users/abdelhaym/minic
onda/bin:/Users/abdelhaym/miniconda/condabin:/usr/local/bin:/System/Cr
yptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.s
ecurity.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.app
le.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple
.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/opt/X
11/bin:/Library/Apple/usr/bin:/Library/TeX/texbin:/usr/local/munki
REQUESTS_CA_BUNDLE=<not set>
SSL_CERT_FILE=<not set>
active environment : base
active env location : /Users/abdelhaym/miniconda
shell level : 1
user config file : /Users/abdelhaym/.condarc
populated config files : /Users/abdelhaym/.condarc
conda version : 23.3.1
conda-build version : not installed
python version : 3.9.16.final.0
virtual packages : __archspec=1=arm64
__osx=14.4.1=0
__unix=0=0
base environment : /Users/abdelhaym/miniconda (writable)
conda av data dir : /Users/abdelhaym/miniconda/etc/conda
conda av metadata url : None
channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
https://conda.anaconda.org/conda-forge/noarch
https://conda.anaconda.org/ncb/osx-arm64
https://conda.anaconda.org/ncb/noarch
https://conda.anaconda.org/nvidia/osx-arm64
https://conda.anaconda.org/nvidia/noarch
https://conda.anaconda.org/apple/osx-arm64
https://conda.anaconda.org/apple/noarch
package cache : /Users/abdelhaym/.conda/pkgs
envs directories : /Users/abdelhaym/.conda/envs
/Users/abdelhaym/miniconda/envs
platform : osx-arm64
user-agent : conda/23.3.1 requests/2.32.3 CPython/3.9.16 Darwin/23.4.0 OSX/14.4.1
UID:GID : 1214968452:140006770
netrc file : None
offline mode : False
Ah, I see. I don't think your institute needs to do that. To avoid licensing issues, all they need to do is block the anaconda channel (https://conda.anaconda.org/anaconda), not the user channels. The above command works with an MPI and our institute - so I guess they block channels more selectively.
Don't think there is much I can do about this atm. I could share an environment file, but even that requires access to the ncb channel.
In the future, we might try to get DAS onto the conda-forge channel, but this is a lengthy process and would make deployment more complicated.
Thanks for checking. I will talk with IT and see if I can find a solution.
Thanks! Would be good for me and other's to know!
In the meantime, I will explore how to get DAS onto conda-forge.
@hummuscience Did you update your ~/.condarc file to remove "defaults" from the channels? (IT sent an email about this in September). That should allow it to get around the firewall and only query conda-forge, if I've understood things correctly.
Thanks for the note. Yes, "defaults" were removed from condarc.
The problem is that DAS needs to query the ncb channel, not just the conda-forge channel. For some reason, that one also seems to be blocked.
@avakiai yes. Defaults is removed from condarc. Did you manage to install DAS on your machine? (@postpop we work in the same institute)
Yes, I recently upgraded my local machine to have the latest version of das. Initially I ran into some issues, but after some fixes from Jan (above), it worked fine. I wasn't aware that they changed anything about the firewall since September, hence I'm a bit puzzled.
In case it solves your problem for now, there is a working das env in the cluster. It's a recent but not the latest version - I imagine that shouldn't make a large difference though.