torchdata not compatible with torch 2.3.0

[mmdanziger]

🐛 Describe the bug

Describe the bug

torchdata does not work with torch 2.3.0 because DILL_AVAILABLE is not available where expected:

Python 3.12.2 (main, Feb  6 2024, 20:19:44) [Clang 15.0.0 (clang-1500.1.0.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import torchdata
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/private/tmp/test/lib/python3.12/site-packages/torchdata/__init__.py", line 9, in <module>
    from . import datapipes
  File "/private/tmp/test/lib/python3.12/site-packages/torchdata/datapipes/__init__.py", line 9, in <module>
    from . import iter, map, utils
  File "/private/tmp/test/lib/python3.12/site-packages/torchdata/datapipes/iter/__init__.py", line 79, in <module>
    from torchdata.datapipes.iter.util.cacheholder import (
  File "/private/tmp/test/lib/python3.12/site-packages/torchdata/datapipes/iter/util/cacheholder.py", line 24, in <module>
    from torch.utils.data.datapipes.utils.common import _check_unpickable_fn, DILL_AVAILABLE
ImportError: cannot import name 'DILL_AVAILABLE' from 'torch.utils.data.datapipes.utils.common' (/private/tmp/test/lib/python3.12/site-packages/torch/utils/data/datapipes/utils/common.py)
>>>

This was fixed in https://github.com/pytorch/pytorch/pull/122616 but the fix did not make it into the torch 2.3.0 release https://github.com/pytorch/pytorch/blob/v2.3.0/torch/utils/data/datapipes/utils/common.py

I would assume 2.3.1 will be ok because the fix is in main. For now though, a clean install of torchdata is broken.

I would recommend adding a torch>=2,!=2.3.0 to setup.py

To Reproduce

pip install torchdata import torchdata

Expected behavior

successful import

Environment

fresh install

Versions

Collecting environment information... PyTorch version: 2.3.0 Is debug build: False CUDA used to build PyTorch: None ROCM used to build PyTorch: N/A

OS: macOS 14.4.1 (arm64) GCC version: Could not collect Clang version: 15.0.0 (clang-1500.3.9.4) CMake version: version 3.27.1 Libc version: N/A

Python version: 3.12.2 (main, Feb 6 2024, 20:19:44) [Clang 15.0.0 (clang-1500.1.0.2.5)] (64-bit runtime) Python platform: macOS-14.4.1-arm64-arm-64bit Is CUDA available: False CUDA runtime version: No CUDA CUDA_MODULE_LOADING set to: N/A GPU models and configuration: No CUDA Nvidia driver version: No CUDA cuDNN version: No CUDA HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True

CPU: Apple M1 Max

Versions of relevant libraries: [pip3] numpy==1.26.4 [pip3] torch==2.3.0 [pip3] torchdata==0.7.1 [conda] Could not collect

[Rhett-Ying]

I encountered the same issue. Is there any ETA to merge the fix into torch 2.3?

[h-vetinari]

We'd also need this as soon as feasible for rolling out pytorch 2.3 support in conda-forge.

[atalman]

This is duplicate of https://github.com/pytorch/pytorch/issues/124907