conda
conda copied to clipboard
conda
Sub-second resolution for repodata.json cache
Checklist
- [X] I added a descriptive title
- [X] I searched open reports and couldn't find a duplicate
What happened?
In conda.testing.solver_helpers.SolverTests.test_nonexistent_deps, the local repodata jsons are updated very quickly, sometimes under one second.
Since the repodata state uses HTTP timestamps, which are tied to HH:MM:SS resolution, we will miss sub-second updates. It looks like we could set the etag to allow sub-second updates.
Conda Info
active environment : base
active env location : /opt/conda
shell level : 1
user config file : /root/.condarc
populated config files : /opt/conda/.condarc
conda version : 24.1.3.dev94+g298ca49ff.d20240409
conda-build version : 24.3.0
python version : 3.12.2.final.0
solver : libmamba (default)
virtual packages : __archspec=1=m1
__conda=24.1.3.dev94+g298ca49ff.d20240409=0
__glibc=2.31=0
__linux=5.15.49=0
__unix=0=0
base environment : /opt/conda (writable)
conda av data dir : /opt/conda/etc/conda
conda av metadata url : None
channel URLs : https://conda.anaconda.org/conda-forge/linux-aarch64
https://conda.anaconda.org/conda-forge/noarch
package cache : /opt/conda/pkgs
/root/.conda/pkgs
envs directories : /opt/conda/envs
/root/.conda/envs
platform : linux-aarch64
user-agent : conda/24.1.3.dev94+g298ca49ff.d20240409 requests/2.31.0 CPython/3.12.2 Linux/5.15.49-linuxkit-pr ubuntu/20.04.6 glibc/2.31 solver/libmamba conda-libmamba-solver/24.1.1.dev49+g38c7f70.d20240408 libmambapy/2.0.0
UID:GID : 0:0
netrc file : None
offline mode : False
Conda Config
==> /opt/conda/.condarc <==
channels:
- conda-forge
Conda list
# packages in environment at /opt/conda:
#
# Name Version Build Channel
_openmp_mutex 4.5 2_gnu conda-forge
anaconda-client 1.12.3 pyhd8ed1ab_0 conda-forge
anaconda-project 0.11.1 pyhd8ed1ab_0 conda-forge
annotated-types 0.6.0 pyhd8ed1ab_0 conda-forge
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
archspec 0.2.3 pyhd8ed1ab_0 conda-forge
attrs 23.2.0 pyh71513ae_0 conda-forge
beautifulsoup4 4.12.3 pyha770c72_0 conda-forge
blinker 1.7.0 pyhd8ed1ab_0 conda-forge
boltons 24.0.0 pyhd8ed1ab_0 conda-forge
boto3 1.34.79 pyhd8ed1ab_0 conda-forge
botocore 1.34.79 pyge310_1234567_0 conda-forge
brotli-python 1.1.0 py312h2aa54b4_1 conda-forge
bzip2 1.0.8 h31becfc_5 conda-forge
c-ares 1.24.0 h31becfc_0 conda-forge
ca-certificates 2024.2.2 hcefe29a_0 conda-forge
cachecontrol 0.14.0 pyhd8ed1ab_0 conda-forge
cachecontrol-with-filecache 0.14.0 pyhd8ed1ab_0 conda-forge
cachy 0.3.0 pyhd8ed1ab_1 conda-forge
certifi 2024.2.2 pyhd8ed1ab_0 conda-forge
cffi 1.16.0 py312hf3c74c0_0 conda-forge
cfgv 3.3.1 pyhd8ed1ab_0 conda-forge
chardet 5.2.0 py312h8025657_1 conda-forge
charset-normalizer 3.3.2 pyhd8ed1ab_0 conda-forge
click 8.1.7 unix_pyh707e725_0 conda-forge
click-default-group 1.2.4 pyhd8ed1ab_0 conda-forge
clikit 0.6.2 pyhd8ed1ab_2 conda-forge
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
conda 24.1.3.dev87+g50d6026a9.d20240408 dev_0 <develop>
conda-build 24.3.0 py312h996f985_1 conda-forge
conda-content-trust 0.2.0 pyhd8ed1ab_0 conda-forge
conda-index 0.4.0 pyhd8ed1ab_0 conda-forge
conda-libmamba-solver 24.1.1.dev49+g38c7f70.d20240408 pypi_0 pypi
conda-lock 2.5.6 pyhd8ed1ab_0 conda-forge
conda-pack 0.7.1 pyhd8ed1ab_0 conda-forge
conda-package-handling 2.2.0 pyh38be061_0 conda-forge
conda-package-streaming 0.9.0 pyhd8ed1ab_0 conda-forge
coverage 7.4.4 py312h9ef2f89_0 conda-forge
cpp-expected 1.1.0 h4c384f3_0 conda-forge
crashtest 0.4.1 pyhd8ed1ab_0 conda-forge
cryptography 42.0.5 py312h6f14deb_0 conda-forge
curl 8.7.1 h4e8248e_0 conda-forge
dbus 1.13.6 h12b9eeb_3 conda-forge
defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge
distlib 0.3.8 pyhd8ed1ab_0 conda-forge
distro 1.9.0 pyhd8ed1ab_0 conda-forge
ensureconda 1.4.4 pyhd8ed1ab_0 conda-forge
exceptiongroup 1.2.0 pyhd8ed1ab_2 conda-forge
expat 2.6.2 h2f0025b_0 conda-forge
filelock 3.13.3 pyhd8ed1ab_0 conda-forge
flask 3.0.2 pyhd8ed1ab_0 conda-forge
fmt 10.2.1 h2a328a1_0 conda-forge
freetype 2.12.1 hf0a5ef3_2 conda-forge
gettext 0.22.5 h2f0025b_2 conda-forge
gettext-tools 0.22.5 h2f0025b_2 conda-forge
git 2.44.0 pl5321he5f4d6e_0 conda-forge
gitdb 4.0.11 pyhd8ed1ab_0 conda-forge
gitpython 3.1.43 pyhd8ed1ab_0 conda-forge
html5lib 1.1 pyh9f0ad1d_0 conda-forge
icu 73.2 h787c7f5_0 conda-forge
identify 2.5.35 pyhd8ed1ab_0 conda-forge
idna 3.6 pyhd8ed1ab_0 conda-forge
importlib-metadata 7.1.0 pyha770c72_0 conda-forge
importlib_metadata 7.1.0 hd8ed1ab_0 conda-forge
importlib_resources 6.4.0 pyhd8ed1ab_0 conda-forge
iniconfig 2.0.0 pyhd8ed1ab_0 conda-forge
itsdangerous 2.1.2 pyhd8ed1ab_0 conda-forge
jaraco.classes 3.4.0 pyhd8ed1ab_0 conda-forge
jaraco.context 4.3.0 pyhd8ed1ab_0 conda-forge
jaraco.functools 4.0.0 pyhd8ed1ab_0 conda-forge
jeepney 0.8.0 pyhd8ed1ab_0 conda-forge
jinja2 3.1.3 pyhd8ed1ab_0 conda-forge
jmespath 1.0.1 pyhd8ed1ab_0 conda-forge
jsonpatch 1.33 pyhd8ed1ab_0 conda-forge
jsonpointer 2.4 py312h996f985_3 conda-forge
jsonschema 4.21.1 pyhd8ed1ab_0 conda-forge
jsonschema-specifications 2023.12.1 pyhd8ed1ab_0 conda-forge
jupyter_core 5.7.2 py312h996f985_0 conda-forge
keyring 25.1.0 pyha804496_0 conda-forge
keyutils 1.6.1 h4e544f5_0 conda-forge
krb5 1.21.2 hc419048_0 conda-forge
lcms2 2.16 h922389a_0 conda-forge
ld_impl_linux-aarch64 2.40 h2d8c526_0 conda-forge
lerc 4.0.0 h4de3ea5_0 conda-forge
libarchive 3.7.2 hd2f85e0_1 conda-forge
libasprintf 0.22.5 h7b6a552_2 conda-forge
libasprintf-devel 0.22.5 h7b6a552_2 conda-forge
libcurl 8.7.1 h4e8248e_0 conda-forge
libdeflate 1.20 h31becfc_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h31becfc_2 conda-forge
libexpat 2.6.2 h2f0025b_0 conda-forge
libffi 3.4.2 h3557bc0_5 conda-forge
libgcc-ng 13.2.0 hf8544c7_3 conda-forge
libgettextpo 0.22.5 h2f0025b_2 conda-forge
libgettextpo-devel 0.22.5 h2f0025b_2 conda-forge
libglib 2.80.0 h9d8fbc1_3 conda-forge
libgomp 13.2.0 hf8544c7_3 conda-forge
libiconv 1.17 h31becfc_2 conda-forge
libjpeg-turbo 3.0.0 h31becfc_1 conda-forge
liblief 0.14.1 h0a1ffab_1 conda-forge
libmamba 2.0.0beta0 h19be64c_0 conda-forge/label/mamba_dev
libmambapy 2.0.0beta0 py312hcab53d1_0 conda-forge/label/mamba_dev
libnghttp2 1.58.0 hb0e430d_1 conda-forge
libnsl 2.0.1 h31becfc_0 conda-forge
libpng 1.6.43 h194ca79_0 conda-forge
libsolv 0.7.28 hd84c7bf_2 conda-forge
libsqlite 3.45.2 h194ca79_0 conda-forge
libssh2 1.11.0 h492db2e_0 conda-forge
libstdcxx-ng 13.2.0 h9a76618_3 conda-forge
libtiff 4.6.0 hf980d43_3 conda-forge
libuuid 2.38.1 hb4cce97_0 conda-forge
libwebp-base 1.3.2 h31becfc_0 conda-forge
libxcb 1.15 h2a766a3_0 conda-forge
libxcrypt 4.4.36 h31becfc_1 conda-forge
libxml2 2.12.3 h3091e33_0 conda-forge
libzlib 1.2.13 h31becfc_5 conda-forge
lz4-c 1.9.4 hd600fc2_0 conda-forge
lzo 2.10 h516909a_1000 conda-forge
mamba 2.0.0beta0 h7eea963_0 conda-forge/label/mamba_dev
markupsafe 2.1.5 py312h9ef2f89_0 conda-forge
menuinst 2.0.2 py312h996f985_0 conda-forge
minio-server 2024.03.10.02.53.48 hcefe29a_0 conda-forge
more-itertools 10.2.0 pyhd8ed1ab_0 conda-forge
msgpack-python 1.0.7 py312h721a97f_0 conda-forge
nbformat 5.10.4 pyhd8ed1ab_0 conda-forge
ncurses 6.4 h0425590_2 conda-forge
nlohmann_json 3.11.3 h2f0025b_0 conda-forge
nodeenv 1.8.0 pyhd8ed1ab_0 conda-forge
openjpeg 2.5.2 h0d9d63b_0 conda-forge
openssl 3.2.1 h31becfc_1 conda-forge
packaging 24.0 pyhd8ed1ab_0 conda-forge
pastel 0.2.1 pyhd8ed1ab_0 conda-forge
patch 2.7.6 hf897c2e_1002 conda-forge
patchelf 0.17.2 h884eca8_0 conda-forge
pcre2 10.43 hd0f9c67_0 conda-forge
perl 5.32.1 7_h31becfc_perl5 conda-forge
pexpect 4.9.0 pyhd8ed1ab_0 conda-forge
pillow 10.3.0 py312h317ddc0_0 conda-forge
pip 24.0 pyhd8ed1ab_0 conda-forge
pkginfo 1.10.0 pyhd8ed1ab_0 conda-forge
pkgutil-resolve-name 1.3.10 pyhd8ed1ab_1 conda-forge
platformdirs 4.2.0 pyhd8ed1ab_0 conda-forge
pluggy 1.4.0 pyhd8ed1ab_0 conda-forge
pre-commit 3.7.0 pyha770c72_0 conda-forge
psutil 5.9.8 py312hdd3e373_0 conda-forge
pthread-stubs 0.4 hb9de7d4_1001 conda-forge
ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge
py-lief 0.14.1 py312h7f10901_1 conda-forge
pybind11-abi 4 hd8ed1ab_3 conda-forge
pycosat 0.6.6 py312hdd3e373_0 conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pydantic 2.6.4 pyhd8ed1ab_0 conda-forge
pydantic-core 2.16.3 py312h3abe38b_0 conda-forge
pylev 1.4.0 pyhd8ed1ab_0 conda-forge
pysocks 1.7.1 pyha2e5f31_6 conda-forge
pytest 8.1.1 pyhd8ed1ab_0 conda-forge
pytest-cov 5.0.0 pyhd8ed1ab_0 conda-forge
pytest-mock 3.14.0 pyhd8ed1ab_0 conda-forge
pytest-rerunfailures 14.0 pyhd8ed1ab_0 conda-forge
pytest-split 0.8.2 pyhd8ed1ab_0 conda-forge
pytest-timeout 2.3.1 pyhd8ed1ab_1 conda-forge
pytest-xprocess 1.0.1 pyhd8ed1ab_0 conda-forge
python 3.12.2 h43d1f9e_0_cpython conda-forge
python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge
python-fastjsonschema 2.19.1 pyhd8ed1ab_0 conda-forge
python-libarchive-c 5.1 py312h996f985_0 conda-forge
python_abi 3.12 4_cp312 conda-forge
pytz 2024.1 pyhd8ed1ab_0 conda-forge
pyyaml 6.0.1 py312hdd3e373_1 conda-forge
readline 8.2 h8fc344f_1 conda-forge
referencing 0.34.0 pyhd8ed1ab_0 conda-forge
reproc 14.2.4.post0 h31becfc_1 conda-forge
reproc-cpp 14.2.4.post0 h2f0025b_1 conda-forge
requests 2.31.0 pyhd8ed1ab_0 conda-forge
requests-toolbelt 1.0.0 pyhd8ed1ab_0 conda-forge
responses 0.25.0 pyhd8ed1ab_0 conda-forge
ripgrep 14.1.0 h1d8f897_0 conda-forge
rpds-py 0.18.0 py312h2a5c9d6_0 conda-forge
ruamel.yaml 0.18.6 py312hdd3e373_0 conda-forge
ruamel.yaml.clib 0.2.8 py312hdd3e373_0 conda-forge
ruamel_yaml 0.15.80 py312hdd3e373_1009 conda-forge
s3transfer 0.10.1 pyhd8ed1ab_0 conda-forge
secretstorage 3.3.3 py312h8025657_2 conda-forge
setuptools 69.2.0 pyhd8ed1ab_0 conda-forge
setuptools-scm 8.0.4 pyhd8ed1ab_1 conda-forge
setuptools_scm 8.0.4 hd8ed1ab_1 conda-forge
simdjson 3.8.0 h2a328a1_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
smmap 5.0.0 pyhd8ed1ab_0 conda-forge
soupsieve 2.5 pyhd8ed1ab_1 conda-forge
spdlog 1.12.0 h6b8df57_2 conda-forge
sqlite 3.45.2 h3b3482f_0 conda-forge
tk 8.6.13 h194ca79_0 conda-forge
toml 0.10.2 pyhd8ed1ab_0 conda-forge
tomli 2.0.1 pyhd8ed1ab_0 conda-forge
tomlkit 0.12.4 pyha770c72_0 conda-forge
toolz 0.12.1 pyhd8ed1ab_0 conda-forge
tornado 6.4 py312h9ef2f89_0 conda-forge
tqdm 4.66.2 pyhd8ed1ab_0 conda-forge
traitlets 5.14.2 pyhd8ed1ab_0 conda-forge
truststore 0.8.0 pyhd8ed1ab_0 conda-forge
types-pyyaml 6.0.12.20240311 pyhd8ed1ab_0 conda-forge
typing-extensions 4.11.0 hd8ed1ab_0 conda-forge
typing_extensions 4.11.0 pyha770c72_0 conda-forge
tzdata 2023c h71feb2d_0 conda-forge
ukkonen 1.0.1 py312h8f0b210_4 conda-forge
urllib3 1.26.18 pyhd8ed1ab_0 conda-forge
virtualenv 20.25.1 pyhd8ed1ab_0 conda-forge
webencodings 0.5.1 pyhd8ed1ab_2 conda-forge
werkzeug 3.0.2 pyhd8ed1ab_0 conda-forge
wheel 0.42.0 pyhd8ed1ab_0 conda-forge
xdoctest 1.1.3 pyhd8ed1ab_0 conda-forge
xorg-libxau 1.0.11 h31becfc_0 conda-forge
xorg-libxdmcp 1.1.3 h3557bc0_0 conda-forge
xz 5.2.6 h9cdd2b7_0 conda-forge
yaml 0.2.5 hf897c2e_2 conda-forge
yaml-cpp 0.8.0 h2f0025b_0 conda-forge
zipp 3.17.0 pyhd8ed1ab_0 conda-forge
zstandard 0.22.0 py312hb120188_0 conda-forge
zstd 1.5.5 h4c53e97_0 conda-forge
Additional Context
Discovered at https://github.com/conda/conda-libmamba-solver/pull/457. For now I'm adding a one-second sleep before the repodata is updated as a workaround, but this might be important for fast changing channels.
Oh, I thought we always used etag only when fetching repodata... in the cache pkgs/cache/*.info.json I see that our CDN channels store etag and mod, but dynamic channels only supply mod with etag stored as the empty string.
I observed this with local channels; e.g. conda create -dnx -c ./path/to/channel. Is it ok to to put hash of the full timestamp (with microsecond resolution) there?
I tried this in some failing tests at https://github.com/conda/conda-libmamba-solver/pull/457 (CONDA_REPODATA_TTL=0 pytest tests/models/test_prefix_graph.py -k libmamba) and it doesn't help. Adding a time.sleep(1) and the beginning of the test does fix it because it ensures the mod time it's different.
Hi there, thank you for your contribution!
This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.
If you would like this issue to remain open please:
- Verify that you can still reproduce the issue at hand
- Comment that the issue is still reproducible and include: - What OS and version you reproduced the issue on - What steps you followed to reproduce the issue
NOTE: If this issue was closed prematurely, please leave a comment.
Thanks!
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}