cudf
cudf copied to clipboard
rapidsai
[BUG] Unexpected OOMs encountered with `read_csv` on WSL2
Describe the bug While testing the cuGraph's UVM notebook, I encountered an OOM error when trying to read in a large (~26 GB) CSV dataset on an RTX 8000 48GB.
Steps/Code to reproduce bug Sorry for the the lengthy reproducer - happy to switch over a more readily available large dataset if possible:
import os
import urllib.request
import cudf
import rmm
rmm.mr.set_current_device_resource(rmm.mr.ManagedMemoryResource())
data_dir = './data/'
if not os.path.exists(data_dir):
print('creating data directory')
os.system('mkdir ./data')
# download the Twitter dataset
base_url = 'https://s3.us-east-2.amazonaws.com/rapidsai-data/cugraph/benchmark/'
fn = 'twitter-2010.csv'
comp = '.gz'
if not os.path.isfile(data_dir+fn):
if not os.path.isfile(data_dir+fn+comp):
print(f'Downloading {base_url+fn+comp} to {data_dir+fn+comp}')
urllib.request.urlretrieve(base_url+fn+comp, data_dir+fn+comp)
print(f'Decompressing {data_dir+fn+comp}...')
os.system('gunzip '+data_dir+fn+comp)
print(f'{data_dir+fn+comp} decompressed!')
else:
print(f'Your data file, {data_dir+fn}, already exists')
# File path, assuming Notebook directory
input_data_path = data_dir+fn
e_list = cudf.read_csv(input_data_path, delimiter=' ', names=['src', 'dst'], dtype=['int32', 'int32'])
The above fails unless nrows is set to something under ~100,000,000:
MemoryError Traceback (most recent call last)
/tmp/ipykernel_880/4119338851.py in <module>
3
4 # CSV reader
----> 5 e_list = cudf.read_csv(input_data_path, delimiter=' ', names=['src', 'dst'], dtype=['int32', 'int32'])
6
7 # Print time
/opt/conda/envs/rapids/lib/python3.9/contextlib.py in inner(*args, **kwds)
77 def inner(*args, **kwds):
78 with self._recreate_cm():
---> 79 return func(*args, **kwds)
80 return inner
81
/opt/conda/envs/rapids/lib/python3.9/site-packages/cudf/io/csv.py in read_csv(filepath_or_buffer, lineterminator, quotechar, quoting, doublequote, header, mangle_dupe_cols, usecols, sep, delimiter, delim_whitespace, skipinitialspace, names, dtype, skipfooter, skiprows, dayfirst, compression, thousands, decimal, true_values, false_values, nrows, byte_range, skip_blank_lines, parse_dates, comment, na_values, keep_default_na, na_filter, prefix, index_col, use_python_file_object, **kwargs)
71 na_values = [na_values]
72
---> 73 return libcudf.csv.read_csv(
74 filepath_or_buffer,
75 lineterminator=lineterminator,
cudf/_lib/csv.pyx in cudf._lib.csv.read_csv()
MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /rapids/rmm/include/rmm/mr/device/managed_memory_resource.hpp:74: cudaErrorMemoryAllocation out of memory
Expected behavior I would expect the CSV dataset to be read in its entirety - the notebook and this code succeed on a standard Linux machine with a V100 32GB (DGX1).
Environment overview (please complete the following information)
- Environment location: Docker
- Method of cuDF install: Docker
docker run --gpus all -p 8888:8888 -p 8787:8787 -p 8786:8786 rapidsai/rapidsai-core-dev-nightly:22.04-cuda11.5-devel-ubuntu20.04-py3.9
Environment details
Click here to see environment details
**git***print_env.sh: 10: [: true: unexpected operator Not inside a git repository
***OS Information*** DISTRIB_ID=Ubuntu DISTRIB_RELEASE=20.04 DISTRIB_CODENAME=focal DISTRIB_DESCRIPTION="Ubuntu 20.04.4 LTS" NAME="Ubuntu" VERSION="20.04.4 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.4 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal Linux 4ab1a0161978 5.10.102.1-microsoft-standard-WSL2 #1 SMP Wed Mar 2 00:30:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux ***GPU Information*** Thu Mar 31 14:45:02 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 510.47.03 Driver Version: 511.65 CUDA Version: 11.6 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Quadro RTX 8000 On | 00000000:15:00.0 Off | Off | | 34% 32C P8 17W / 260W | 823MiB / 49152MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ | 1 Quadro RTX 8000 On | 00000000:2D:00.0 On | Off | | 35% 61C P0 72W / 260W | 1906MiB / 49152MiB | 15% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 804 C /python3.9 N/A | | 1 N/A N/A 804 C /python3.9 N/A | +-----------------------------------------------------------------------------+ ***CPU*** Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 12 On-line CPU(s) list: 0-11 Thread(s) per core: 2 Core(s) per socket: 6 Socket(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz Stepping: 4 CPU MHz: 3391.499 BogoMIPS: 6782.99 Virtualization: VT-x Hypervisor vendor: Microsoft Virtualization type: full L1d cache: 192 KiB L1i cache: 192 KiB L2 cache: 6 MiB L3 cache: 19.3 MiB Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable Vulnerability Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown Vulnerability Meltdown: Mitigation; PTI Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves flush_l1d arch_capabilities ***CMake*** /opt/conda/envs/rapids/bin/cmake cmake version 3.20.5 CMake suite maintained and supported by Kitware (kitware.com/cmake). ***g++*** /usr/bin/g++ g++ (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0 Copyright (C) 2019 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. ***nvcc*** /usr/local/cuda/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2021 NVIDIA Corporation Built on Thu_Nov_18_09:45:30_PST_2021 Cuda compilation tools, release 11.5, V11.5.119 Build cuda_11.5.r11.5/compiler.30672275_0 ***Python*** /opt/conda/envs/rapids/bin/python Python 3.9.12 ***Environment Variables*** PATH : /opt/conda/envs/rapids/bin:/opt/conda/condabin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LD_LIBRARY_PATH : /usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/lib:/opt/conda/envs/rapids/lib NUMBAPRO_NVVM : NUMBAPRO_LIBDEVICE : CONDA_PREFIX : /opt/conda/envs/rapids PYTHON_PATH : ***conda packages*** conda is /opt/conda/envs/rapids/bin/conda /opt/conda/envs/rapids/bin/conda # packages in environment at /opt/conda/envs/rapids: # # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 1_gnu conda-forge abseil-cpp 20211102.0 h27087fc_1 conda-forge aiobotocore 2.1.0 pyhd8ed1ab_0 conda-forge aiohttp 3.8.1 py39h3811e60_0 conda-forge aioitertools 0.10.0 pyhd8ed1ab_0 conda-forge aiosignal 1.2.0 pyhd8ed1ab_0 conda-forge alabaster 0.7.12 py_0 conda-forge anyio 3.5.0 py39hf3d152e_0 conda-forge appdirs 1.4.4 pyh9f0ad1d_0 conda-forge argon2-cffi 21.3.0 pyhd8ed1ab_0 conda-forge argon2-cffi-bindings 21.2.0 py39h3811e60_1 conda-forge arrow-cpp 6.0.1 py39hbfed05e_11_cuda conda-forge arrow-cpp-proc 3.0.0 cuda conda-forge asn1crypto 1.5.1 pyhd8ed1ab_0 conda-forge asvdb 0.4.2 g90e8f2c_40 rapidsai-nightly async-timeout 4.0.2 pyhd8ed1ab_0 conda-forge atk-1.0 2.36.0 h3371d22_4 conda-forge attrs 21.4.0 pyhd8ed1ab_0 conda-forge autoconf 2.69 pl5321hd708f79_11 conda-forge automake 1.16.5 pl5321ha770c72_0 conda-forge aws-c-cal 0.5.11 h95a6274_0 conda-forge aws-c-common 0.6.2 h7f98852_0 conda-forge aws-c-event-stream 0.2.7 h3541f99_13 conda-forge aws-c-io 0.10.5 hfb6a706_0 conda-forge aws-checksums 0.1.11 ha31a3da_7 conda-forge aws-sam-translator 1.43.0 pyhd8ed1ab_0 conda-forge aws-sdk-cpp 1.8.186 hb4091e7_3 conda-forge aws-xray-sdk 2.9.0 pyhd8ed1ab_0 conda-forge babel 2.9.1 pyh44b312d_0 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 py_2 conda-forge backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge beautifulsoup4 4.10.0 pyha770c72_0 conda-forge benchmark 1.5.1 he1b5a44_2 conda-forge binutils_impl_linux-64 2.36.1 h193b22a_2 conda-forge black 19.10b0 py_4 conda-forge blas 2.105 netlib conda-forge blas-devel 3.9.0 5_netlib conda-forge bleach 4.1.0 pyhd8ed1ab_0 conda-forge blinker 1.4 py_1 conda-forge blosc 1.21.0 h9c3ff4c_0 conda-forge bokeh 2.4.2 py39hf3d152e_0 conda-forge boost 1.72.0 py39ha90915f_1 conda-forge boost-cpp 1.72.0 h312852a_5 conda-forge boto3 1.20.24 pyhd8ed1ab_0 conda-forge botocore 1.23.24 pyhd8ed1ab_0 conda-forge breathe 4.33.0 pyhd8ed1ab_0 conda-forge brotli 1.0.9 h7f98852_6 conda-forge brotli-bin 1.0.9 h7f98852_6 conda-forge brotlipy 0.7.0 py39h3811e60_1003 conda-forge brunsli 0.1 h9c3ff4c_0 conda-forge bzip2 1.0.8 h7f98852_4 conda-forge c-ares 1.18.1 h7f98852_0 conda-forge c-blosc2 2.0.4 h5f21a17_1 conda-forge ca-certificates 2021.10.8 ha878542_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge cachetools 5.0.0 pyhd8ed1ab_0 conda-forge cairo 1.16.0 h6cf1ce9_1008 conda-forge certifi 2021.10.8 py39hf3d152e_1 conda-forge cffi 1.15.0 py39h4bc2ebd_0 conda-forge cfgv 3.3.1 pyhd8ed1ab_0 conda-forge cfitsio 3.470 hb418390_7 conda-forge cfn-lint 0.54.2 py39hf3d152e_0 conda-forge chardet 4.0.0 py39hf3d152e_2 conda-forge charls 2.2.0 h9c3ff4c_0 conda-forge charset-normalizer 2.0.12 pyhd8ed1ab_0 conda-forge clang 11.1.0 ha770c72_1 conda-forge clang-11 11.1.0 default_ha53f305_1 conda-forge clang-tools 11.1.0 default_ha53f305_1 conda-forge clangxx 11.1.0 default_ha53f305_1 conda-forge click 7.1.2 pyh9f0ad1d_0 conda-forge click-plugins 1.1.1 py_0 conda-forge cligj 0.7.2 pyhd8ed1ab_1 conda-forge cloudpickle 2.0.0 pyhd8ed1ab_0 conda-forge cmake 3.20.5 h8897547_0 conda-forge cmake-format 0.6.11 pyh9f0ad1d_0 conda-forge cmake_setuptools 0.1.3 py_0 rapidsai-nightly cmarkgfm 0.8.0 py39hb9d737c_0 conda-forge colorama 0.4.4 pyh9f0ad1d_0 conda-forge colorcet 3.0.0 pyhd8ed1ab_0 conda-forge commonmark 0.9.1 py_0 conda-forge conda 4.10.3 py39hf3d152e_4 conda-forge conda-build 3.21.7 py39hf3d152e_0 conda-forge conda-package-handling 1.8.0 py39hb9d737c_0 conda-forge conda-verify 3.1.1 py39hf3d152e_1004 conda-forge coverage 6.3.2 py39hb9d737c_1 conda-forge cryptography 36.0.2 py39hd97740a_0 conda-forge cuda-python 11.6.1 py39h3fd9d12_0 nvidia cudatoolkit 11.5.1 hcf5317a_9 nvidia cudf 0+untagged.1.gc71fe1b pypi_0 pypi cudf-kafka 0+untagged.1.gc71fe1b pypi_0 pypi cugraph 0+untagged.1.g87be0b3 pypi_0 pypi cuml 0+untagged.1.g3798925 pypi_0 pypi cupy 10.2.0 py39hc3c280e_0 conda-forge curl 7.82.0 h7bff187_0 conda-forge cusignal 0+untagged.1.g8878bf7 pypi_0 pypi cuspatial 0+untagged.1.g3637da5 pypi_0 pypi cuxfilter 0+untagged.1.g53c9564 pypi_0 pypi cycler 0.11.0 pyhd8ed1ab_0 conda-forge cyrus-sasl 2.1.27 h230043b_5 conda-forge cython 0.29.28 py39he80948d_0 conda-forge cytoolz 0.11.2 py39h3811e60_1 conda-forge dask 2022.3.0 pyhd8ed1ab_0 conda-forge dask-core 2022.3.0 pyhd8ed1ab_0 conda-forge dask-cudf 0+untagged.1.gc71fe1b pypi_0 pypi dask-glm 0.2.0 py_1 conda-forge dask-labextension 5.2.0 pyhd8ed1ab_0 conda-forge dask-ml 1.9.0 pyhd8ed1ab_0 conda-forge dataclasses 0.8 pyhc8e2a94_3 conda-forge datashader 0.13.0 pyh6c4a22f_0 conda-forge datashape 0.5.4 py_1 conda-forge dbus 1.13.6 h5008d03_3 conda-forge debugpy 1.5.1 py39he80948d_0 conda-forge decorator 5.1.1 pyhd8ed1ab_0 conda-forge defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge distlib 0.3.4 pyhd8ed1ab_0 conda-forge distributed 2022.3.0 pyhd8ed1ab_0 conda-forge distro 1.6.0 pyhd8ed1ab_0 conda-forge dlpack 0.5 h9c3ff4c_0 conda-forge docker-py 5.0.3 py39hf3d152e_2 conda-forge docker-pycreds 0.4.0 py_0 conda-forge docutils 0.17.1 py39hf3d152e_1 conda-forge double-conversion 3.1.5 h9c3ff4c_2 conda-forge doxygen 1.8.20 had0d8f1_0 conda-forge ecdsa 0.17.0 pyhd8ed1ab_0 conda-forge entrypoints 0.4 pyhd8ed1ab_0 conda-forge execnet 1.9.0 pyhd8ed1ab_0 conda-forge expat 2.4.7 h27087fc_0 conda-forge faiss-proc 1.0.0 cuda rapidsai fastavro 1.4.10 py39hb9d737c_0 conda-forge fastrlock 0.8 py39he80948d_1 conda-forge feather-format 0.4.1 pyh9f0ad1d_0 conda-forge filelock 3.6.0 pyhd8ed1ab_0 conda-forge filterpy 1.4.5 py_1 conda-forge fiona 1.8.20 py39h427c1bf_1 conda-forge flake8 3.8.4 py_0 conda-forge flask 2.0.3 pyhd8ed1ab_0 conda-forge flask_cors 3.0.10 pyhd3deb0d_0 conda-forge flit-core 3.7.1 pyhd8ed1ab_0 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 hab24e00_0 conda-forge fontconfig 2.13.96 h8e229c2_2 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.31.2 py39hb9d737c_0 conda-forge freetype 2.10.4 h0708190_1 conda-forge freexl 1.0.6 h7f98852_0 conda-forge fribidi 1.0.10 h36c2ea0_0 conda-forge frozenlist 1.3.0 py39h3811e60_0 conda-forge fsspec 2022.2.0 pyhd8ed1ab_0 conda-forge future 0.18.2 py39hf3d152e_4 conda-forge gcc_impl_linux-64 11.2.0 h82a94d6_14 conda-forge gcsfs 2022.2.0 pyhd8ed1ab_0 conda-forge gdal 3.3.1 py39h218ed2d_3 conda-forge gdk-pixbuf 2.42.8 hff1cb4f_0 conda-forge geopandas 0.9.0 pyhd8ed1ab_1 conda-forge geopandas-base 0.9.0 pyhd8ed1ab_1 conda-forge geos 3.9.1 h9c3ff4c_2 conda-forge geotiff 1.6.0 h4f31c25_6 conda-forge gettext 0.19.8.1 h73d1719_1008 conda-forge gflags 2.2.2 he1b5a44_1004 conda-forge giflib 5.2.1 h36c2ea0_2 conda-forge git 2.35.1 pl5321h36853c3_0 conda-forge git-lfs 3.1.2 ha770c72_0 conda-forge glib 2.70.2 h780b84a_4 conda-forge glib-tools 2.70.2 h780b84a_4 conda-forge glob2 0.7 py_0 conda-forge glog 0.5.0 h48cff8f_0 conda-forge gmock 1.10.0 h4bd325d_7 conda-forge gmp 6.2.1 h58526e2_0 conda-forge google-api-core 2.5.0 pyhd8ed1ab_0 conda-forge google-auth 2.6.2 pyh6c4a22f_0 conda-forge google-auth-oauthlib 0.5.1 pyhd8ed1ab_0 conda-forge google-cloud-core 2.2.2 pyh6c4a22f_0 conda-forge google-cloud-storage 2.1.0 pyh6c4a22f_0 conda-forge google-crc32c 1.1.2 py39hb81f231_2 conda-forge google-resumable-media 2.1.0 pyh6c4a22f_0 conda-forge googleapis-common-protos 1.56.0 py39hf3d152e_0 conda-forge gpuci-tools 0.3.1 12 gpuci graphite2 1.3.13 h58526e2_1001 conda-forge graphql-core 3.2.0 pyhd8ed1ab_0 conda-forge graphviz 2.50.0 h85b4f2f_1 conda-forge grpc-cpp 1.44.0 h3d78c48_1 conda-forge grpcio 1.45.0 py39h0f497a6_0 conda-forge gtest 1.10.0 h4bd325d_7 conda-forge gtk2 2.24.33 h539f30e_1 conda-forge gts 0.7.6 h64030ff_2 conda-forge h5py 3.6.0 nompi_py39h7e08c79_100 conda-forge harfbuzz 3.1.1 h83ec7ef_0 conda-forge hdbscan 0.8.28 py39hce5d2b2_1 conda-forge hdf4 4.2.15 h10796ff_3 conda-forge hdf5 1.12.1 nompi_h2386368_104 conda-forge heapdict 1.0.1 py_0 conda-forge holoviews 1.14.6 pyhd8ed1ab_0 conda-forge html5lib 1.1 pyh9f0ad1d_0 conda-forge httpretty 1.1.4 pyhd8ed1ab_0 conda-forge huggingface_hub 0.4.0 pyhd8ed1ab_0 conda-forge hypothesis 6.39.5 pyhd8ed1ab_0 conda-forge icu 68.2 h9c3ff4c_0 conda-forge identify 2.4.12 pyhd8ed1ab_0 conda-forge idna 3.3 pyhd8ed1ab_0 conda-forge imagecodecs 2021.8.26 py39h44211f0_1 conda-forge imageio 2.16.1 pyhcf75d05_0 conda-forge imagesize 1.3.0 pyhd8ed1ab_0 conda-forge importlib-metadata 4.11.3 py39hf3d152e_0 conda-forge importlib_metadata 4.11.3 hd8ed1ab_0 conda-forge iniconfig 1.1.1 pyh9f0ad1d_0 conda-forge ipykernel 6.9.2 py39hef51801_0 conda-forge ipython 7.31.1 py39hf3d152e_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge ipywidgets 7.7.0 pyhd8ed1ab_0 conda-forge isort 5.6.4 py_0 conda-forge itsdangerous 2.1.2 pyhd8ed1ab_0 conda-forge jbig 2.1 h7f98852_2003 conda-forge jedi 0.18.1 py39hf3d152e_0 conda-forge jeepney 0.7.1 pyhd8ed1ab_0 conda-forge jinja2 3.1.1 pyhd8ed1ab_0 conda-forge jmespath 0.10.0 pyh9f0ad1d_0 conda-forge joblib 1.1.0 pyhd8ed1ab_0 conda-forge jpeg 9e h7f98852_0 conda-forge json-c 0.15 h98cffda_0 conda-forge json5 0.9.5 pyh9f0ad1d_0 conda-forge jsondiff 1.3.1 pyhd8ed1ab_0 conda-forge jsonpatch 1.32 pyhd8ed1ab_0 conda-forge jsonpointer 2.0 py_0 conda-forge jsonschema 3.2.0 pyhd8ed1ab_3 conda-forge junit-xml 1.9 pyh9f0ad1d_0 conda-forge jupyter-packaging 0.7.12 pyhd8ed1ab_0 conda-forge jupyter-server-proxy 3.2.1 pyhd8ed1ab_0 conda-forge jupyter_client 7.1.2 pyhd8ed1ab_0 conda-forge jupyter_core 4.9.2 py39hf3d152e_0 conda-forge jupyter_server 1.15.6 pyhd8ed1ab_1 conda-forge jupyter_sphinx 0.3.2 py39hf3d152e_0 conda-forge jupyterlab 3.3.2 pyhd8ed1ab_0 conda-forge jupyterlab-favorites 3.0.0 pyhd8ed1ab_0 conda-forge jupyterlab-nvdashboard 0.7.0a220307 py_21 rapidsai-nightly jupyterlab_pygments 0.1.2 pyh9f0ad1d_0 conda-forge jupyterlab_server 2.11.0 pyhd8ed1ab_0 conda-forge jupyterlab_widgets 1.1.0 pyhd8ed1ab_0 conda-forge jxrlib 1.1 h7f98852_2 conda-forge kealib 1.4.14 h87e4c3c_3 conda-forge kernel-headers_linux-64 2.6.32 he073ed8_15 conda-forge keyring 23.4.0 py39hf3d152e_2 conda-forge keyutils 1.6.1 h166bdaf_0 conda-forge kiwisolver 1.4.0 py39hf939315_0 conda-forge krb5 1.19.3 h3790be6_0 conda-forge lapack 3.9.0 netlib conda-forge lcms2 2.12 hddcbb42_0 conda-forge ld_impl_linux-64 2.36.1 hea4e1c9_2 conda-forge lerc 3.0 h9c3ff4c_0 conda-forge libaec 1.0.6 h9c3ff4c_0 conda-forge libarchive 3.5.2 hccf745f_1 conda-forge libblas 3.9.0 5_h92ddd45_netlib conda-forge libbrotlicommon 1.0.9 h7f98852_6 conda-forge libbrotlidec 1.0.9 h7f98852_6 conda-forge libbrotlienc 1.0.9 h7f98852_6 conda-forge libcblas 3.9.0 5_h92ddd45_netlib conda-forge libclang-cpp11.1 11.1.0 default_ha53f305_1 conda-forge libcrc32c 1.1.2 h9c3ff4c_0 conda-forge libcugraphops 22.04.00a220329 cuda11_g91ab46e_30 rapidsai-nightly libcumlprims 22.04.00a220324 cuda11_g99e8d8f_15 rapidsai-nightly libcurl 7.82.0 h7bff187_0 conda-forge libcusolver 11.3.4.124 h33c3c4e_0 nvidia libcypher-parser 0.6.2 1 rapidsai-nightly libdap4 3.20.6 hd7c4107_2 conda-forge libdeflate 1.8 h7f98852_0 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 h516909a_1 conda-forge libevent 2.1.10 h9b69904_4 conda-forge libfaiss 1.7.0 cuda112h5bea7ad_8_cuda conda-forge libffi 3.4.2 h7f98852_5 conda-forge libgcc-devel_linux-64 11.2.0 h0952999_14 conda-forge libgcc-ng 11.2.0 h1d223b6_14 conda-forge libgcrypt 1.10.0 h7f98852_0 conda-forge libgd 2.3.3 h6ad9fb6_0 conda-forge libgdal 3.3.1 h6214c1d_3 conda-forge libgfortran-ng 11.2.0 h69a702a_14 conda-forge libgfortran5 11.2.0 h5c6108e_14 conda-forge libglib 2.70.2 h174f98d_4 conda-forge libgomp 11.2.0 h1d223b6_14 conda-forge libgpg-error 1.44 h9eb791d_0 conda-forge libgsasl 1.10.0 h5b4c23d_0 conda-forge libhwloc 2.3.0 h5e5b7d1_1 conda-forge libiconv 1.16 h516909a_0 conda-forge libkml 1.3.0 hd79254b_1012 conda-forge liblapack 3.9.0 5_h92ddd45_netlib conda-forge liblapacke 3.9.0 5_h92ddd45_netlib conda-forge liblief 0.11.5 h9c3ff4c_1 conda-forge libllvm11 11.1.0 hf817b99_3 conda-forge libnetcdf 4.8.1 nompi_hb3fd0d9_101 conda-forge libnghttp2 1.47.0 h727a467_0 conda-forge libnsl 2.0.0 h7f98852_0 conda-forge libntlm 1.4 h7f98852_1002 conda-forge libpng 1.6.37 h21135ba_2 conda-forge libpq 13.5 hd57d9b9_1 conda-forge libprotobuf 3.19.4 h780b84a_0 conda-forge librdkafka 1.7.0 hc49e61c_1 conda-forge librmm 22.04.00a220329 cuda11_g220ba88_44 rapidsai-nightly librsvg 2.52.5 hc3c00ef_1 conda-forge librttopo 1.1.0 h1185371_6 conda-forge libsanitizer 11.2.0 he4da1e4_14 conda-forge libsodium 1.0.18 h36c2ea0_1 conda-forge libspatialindex 1.9.3 h9c3ff4c_4 conda-forge libspatialite 5.0.1 h8694cbe_6 conda-forge libssh2 1.10.0 ha56f1ee_2 conda-forge libstdcxx-ng 11.2.0 he4da1e4_14 conda-forge libthrift 0.15.0 he6d91bd_1 conda-forge libtiff 4.3.0 h6f004c6_2 conda-forge libtmglib 3.9.0 5_h92ddd45_netlib conda-forge libtool 2.4.6 h9c3ff4c_1008 conda-forge libutf8proc 2.7.0 h7f98852_0 conda-forge libuuid 2.32.1 h7f98852_1000 conda-forge libuv 1.43.0 h7f98852_0 conda-forge libwebp 1.2.2 h3452ae3_0 conda-forge libwebp-base 1.2.2 h7f98852_1 conda-forge libxcb 1.13 h7f98852_1004 conda-forge libxml2 2.9.12 h72842e0_0 conda-forge libzip 1.8.0 h4de3113_1 conda-forge libzlib 1.2.11 h166bdaf_1014 conda-forge libzopfli 1.0.3 h9c3ff4c_0 conda-forge lightgbm 3.3.2 py39he80948d_0 conda-forge llvmlite 0.38.0 py39h1bbdace_0 conda-forge locket 0.2.1 pypi_0 pypi lz4-c 1.9.3 h9c3ff4c_1 conda-forge lzo 2.10 h516909a_1000 conda-forge m4 1.4.18 h516909a_1001 conda-forge make 4.3 hd18ef5c_1 conda-forge mapclassify 2.4.3 pyhd8ed1ab_0 conda-forge markdown 3.3.6 pyhd8ed1ab_0 conda-forge markupsafe 2.1.1 py39hb9d737c_0 conda-forge matplotlib-base 3.5.1 py39h2fa2bec_0 conda-forge matplotlib-inline 0.1.3 pyhd8ed1ab_0 conda-forge mccabe 0.6.1 py_1 conda-forge mimesis 4.0.0 pyh9f0ad1d_0 conda-forge mistune 0.8.4 py39h3811e60_1005 conda-forge mock 4.0.3 py39hf3d152e_2 conda-forge moto 3.1.2 pyhd8ed1ab_0 conda-forge msgpack-python 1.0.3 py39h1a9c180_0 conda-forge multidict 6.0.2 py39h3811e60_0 conda-forge multipledispatch 0.6.0 py_0 conda-forge munch 2.5.0 py_0 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge mypy 0.782 py_0 conda-forge mypy_extensions 0.4.3 py39hf3d152e_4 conda-forge nbclassic 0.3.7 pyhd8ed1ab_0 conda-forge nbclient 0.5.13 pyhd8ed1ab_0 conda-forge nbconvert 6.4.4 py39hf3d152e_0 conda-forge nbformat 5.2.0 pyhd8ed1ab_0 conda-forge nbsphinx 0.8.8 pyhd8ed1ab_0 conda-forge nccl 2.12.7.1 h0800d71_0 conda-forge ncurses 6.3 h9c3ff4c_0 conda-forge nest-asyncio 1.5.4 pyhd8ed1ab_0 conda-forge networkx 2.6.3 pyhd8ed1ab_1 conda-forge ninja 1.10.2 h4bd325d_1 conda-forge nltk 3.6.7 pyhd8ed1ab_0 conda-forge nodeenv 1.6.0 pyhd8ed1ab_0 conda-forge nodejs 14.18.3 h92b4a50_1 conda-forge notebook 6.4.10 pyha770c72_0 conda-forge notebook-shim 0.1.0 pyhd8ed1ab_0 conda-forge numba 0.55.1 py39h56b8d98_0 conda-forge numpy 1.21.5 py39haac66dc_0 conda-forge numpydoc 1.2 pyhd8ed1ab_0 conda-forge nvtx 0.2.3 py39h3811e60_1 conda-forge oauthlib 3.2.0 pyhd8ed1ab_0 conda-forge openjpeg 2.4.0 hb52868f_1 conda-forge openslide 3.4.1 h978ee9a_4 conda-forge openssl 1.1.1n h166bdaf_0 conda-forge orc 1.7.3 h1be678f_0 conda-forge packaging 21.3 pyhd8ed1ab_0 conda-forge pandas 1.3.5 py39hde0f152_0 conda-forge pandoc 1.19.2 0 conda-forge pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge panel 0.12.6 pyhd8ed1ab_0 conda-forge pango 1.48.10 h54213e6_2 conda-forge param 1.12.0 pyh6c4a22f_0 conda-forge parquet-cpp 1.5.1 2 conda-forge parso 0.8.3 pyhd8ed1ab_0 conda-forge partd 1.2.0 pyhd8ed1ab_0 conda-forge patchelf 0.14.5 h58526e2_0 conda-forge pathspec 0.9.0 pyhd8ed1ab_0 conda-forge patsy 0.5.2 pyhd8ed1ab_0 conda-forge pcre 8.45 h9c3ff4c_0 conda-forge pcre2 10.37 h032f7d1_0 conda-forge perl 5.32.1 2_h7f98852_perl5 conda-forge pexpect 4.8.0 pyh9f0ad1d_2 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 9.0.1 py39hae2aec6_2 conda-forge pip 22.0.4 pyhd8ed1ab_0 conda-forge pixman 0.40.0 h36c2ea0_0 conda-forge pkg-config 0.29.2 h36c2ea0_1008 conda-forge pkginfo 1.8.2 pyhd8ed1ab_0 conda-forge platformdirs 2.5.1 pyhd8ed1ab_0 conda-forge pluggy 1.0.0 py39hf3d152e_2 conda-forge poppler 21.03.0 h93df280_0 conda-forge poppler-data 0.4.11 hd8ed1ab_0 conda-forge postgresql 13.5 h2510834_1 conda-forge pre-commit 2.17.0 py39hf3d152e_0 conda-forge proj 8.0.1 h277dcde_0 conda-forge prometheus_client 0.13.1 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.27 pyha770c72_0 conda-forge protobuf 3.19.4 py39he80948d_0 conda-forge psutil 5.9.0 py39h3811e60_0 conda-forge pthread-stubs 0.4 h36c2ea0_1001 conda-forge ptxcompiler 0.2.0 py39h107f55c_0 rapidsai-nightly ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge py 1.11.0 pyh6c4a22f_0 conda-forge py-cpuinfo 8.0.0 pyhd8ed1ab_0 conda-forge py-lief 0.11.5 py39he80948d_1 conda-forge pyarrow 6.0.1 py39h1ed2e5d_11_cuda conda-forge pyasn1 0.4.8 py_0 conda-forge pyasn1-modules 0.2.7 py_0 conda-forge pycodestyle 2.6.0 pyh9f0ad1d_0 conda-forge pycosat 0.6.3 py39h3811e60_1009 conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pyct 0.4.6 py_0 conda-forge pyct-core 0.4.6 py_0 conda-forge pydata-sphinx-theme 0.8.1 pyhd8ed1ab_0 conda-forge pydeck 0.5.0 pyh9f0ad1d_0 conda-forge pydocstyle 6.1.1 pyhd8ed1ab_0 conda-forge pyee 8.1.0 pyhd8ed1ab_0 conda-forge pyflakes 2.2.0 pyh9f0ad1d_0 conda-forge pygal 3.0.0 pypi_0 pypi pygments 2.11.2 pyhd8ed1ab_0 conda-forge pyjwt 2.3.0 pyhd8ed1ab_1 conda-forge pylibcugraph 0+untagged.1.g87be0b3 pypi_0 pypi pylibraft 0+untagged.1.g6aa9ac8 pypi_0 pypi pynndescent 0.5.6 pyh6c4a22f_0 conda-forge pynvml 11.4.1 pyhd8ed1ab_0 conda-forge pyopenssl 22.0.0 pyhd8ed1ab_0 conda-forge pyorc 0.6.0 py39h40ac9ea_0 conda-forge pyparsing 3.0.7 pyhd8ed1ab_0 conda-forge pyppeteer 1.0.2 pyhd8ed1ab_0 conda-forge pyproj 3.1.0 py39ha9a7ae0_4 conda-forge pyrsistent 0.18.1 py39h3811e60_0 conda-forge pysocks 1.7.1 py39hf3d152e_4 conda-forge pytest 7.1.1 py39hf3d152e_0 conda-forge pytest-asyncio 0.12.0 py39hde42818_2 conda-forge pytest-benchmark 3.4.1 pyhd8ed1ab_0 conda-forge pytest-cov 3.0.0 pyhd8ed1ab_0 conda-forge pytest-forked 1.4.0 pyhd8ed1ab_0 conda-forge pytest-timeout 2.1.0 pyhd8ed1ab_0 conda-forge pytest-xdist 2.5.0 pyhd8ed1ab_0 conda-forge python 3.9.12 h9a8a25e_1_cpython conda-forge python-confluent-kafka 1.7.0 py39h3811e60_2 conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-jose 3.3.0 pyh6c4a22f_1 conda-forge python-libarchive-c 4.0 py39hf3d152e_0 conda-forge python-louvain 0.15 pyhd8ed1ab_1 conda-forge python-snappy 0.6.0 py39h300dd49_1 conda-forge python_abi 3.9 2_cp39 conda-forge pytz 2022.1 pyhd8ed1ab_0 conda-forge pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge pyviz_comms 2.1.0 pyhd8ed1ab_0 conda-forge pywavelets 1.3.0 py39hd257fcd_0 conda-forge pyyaml 6.0 py39h3811e60_3 conda-forge pyzmq 22.3.0 py39h37b5a0c_1 conda-forge raft 0+untagged.1.g6aa9ac8 pypi_0 pypi rapidjson 1.1.0 he1b5a44_1002 conda-forge re2 2022.02.01 h9c3ff4c_0 conda-forge readline 8.1 h46c0cb4_0 conda-forge readme_renderer 27.0 pyh9f0ad1d_0 conda-forge recommonmark 0.7.1 pyhd8ed1ab_0 conda-forge regex 2022.3.15 py39hb9d737c_0 conda-forge requests 2.27.1 pyhd8ed1ab_0 conda-forge requests-oauthlib 1.3.1 pyhd8ed1ab_0 conda-forge requests-toolbelt 0.9.1 py_0 conda-forge responses 0.20.0 pyhd8ed1ab_0 conda-forge rfc3986 2.0.0 pyhd8ed1ab_0 conda-forge rhash 1.4.1 h7f98852_0 conda-forge ripgrep 13.0.0 h2f28480_2 conda-forge rmm 0+untagged.1.g220ba88 pypi_0 pypi rsa 4.8 pyhd8ed1ab_0 conda-forge rtree 0.9.7 py39hb102c33_3 conda-forge ruamel_yaml 0.15.80 py39h3811e60_1006 conda-forge s2n 1.0.10 h9b69904_0 conda-forge s3fs 2022.2.0 pyhd8ed1ab_0 conda-forge s3transfer 0.5.2 pyhd8ed1ab_0 conda-forge sacremoses 0.0.49 pyhd8ed1ab_0 conda-forge sccache 0.2.15 h9b69904_1 conda-forge scikit-build 0.13.1 pyhca92ed8_0 conda-forge scikit-image 0.19.2 py39hde0f152_0 conda-forge scikit-learn 0.24.2 py39h7c5d8c9_1 conda-forge scipy 1.6.0 py39hee8e79c_0 conda-forge seaborn 0.11.2 hd8ed1ab_0 conda-forge seaborn-base 0.11.2 pyhd8ed1ab_0 conda-forge secretstorage 3.3.1 py39hf3d152e_1 conda-forge send2trash 1.8.0 pyhd8ed1ab_0 conda-forge setuptools 59.8.0 py39hf3d152e_0 conda-forge shapely 1.8.0 py39ha61afbd_0 conda-forge shellcheck 0.8.0 ha770c72_0 conda-forge simpervisor 0.4 pyhd8ed1ab_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge snappy 1.1.8 he1b5a44_3 conda-forge sniffio 1.2.0 py39hf3d152e_2 conda-forge snowballstemmer 2.2.0 pyhd8ed1ab_0 conda-forge sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge soupsieve 2.3.1 pyhd8ed1ab_0 conda-forge spdlog 1.8.5 h4bd325d_1 conda-forge sphinx 4.4.0 pyh6c4a22f_1 conda-forge sphinx-click 3.1.0 pyhd8ed1ab_0 conda-forge sphinx-copybutton 0.5.0 pyhd8ed1ab_0 conda-forge sphinx-markdown-tables 0.0.15 pyhd3deb0d_0 conda-forge sphinx_rtd_theme 1.0.0 pyhd8ed1ab_0 conda-forge sphinxcontrib-applehelp 1.0.2 py_0 conda-forge sphinxcontrib-devhelp 1.0.2 py_0 conda-forge sphinxcontrib-htmlhelp 2.0.0 pyhd8ed1ab_0 conda-forge sphinxcontrib-jsmath 1.0.1 py_0 conda-forge sphinxcontrib-qthelp 1.0.3 py_0 conda-forge sphinxcontrib-serializinghtml 1.1.5 pyhd8ed1ab_1 conda-forge sphinxcontrib-websupport 1.2.4 pyhd8ed1ab_1 conda-forge sqlite 3.37.1 h4ff8645_0 conda-forge sshpubkeys 3.1.0 py_0 conda-forge statsmodels 0.13.2 py39hce5d2b2_0 conda-forge streamz 0.6.3 pyh6c4a22f_0 conda-forge sysroot_linux-64 2.12 he073ed8_15 conda-forge tbb 2021.5.0 h4bd325d_0 conda-forge tblib 1.7.0 pyhd8ed1ab_0 conda-forge terminado 0.13.3 py39hf3d152e_0 conda-forge testpath 0.6.0 pyhd8ed1ab_0 conda-forge threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge tifffile 2021.11.2 pyhd8ed1ab_0 conda-forge tiledb 2.3.4 he87e0bf_0 conda-forge tk 8.6.12 h27826a3_0 conda-forge tokenizers 0.10.3 py39hd6d55de_1 conda-forge toml 0.10.2 pyhd8ed1ab_0 conda-forge tomli 2.0.1 pyhd8ed1ab_0 conda-forge toolz 0.11.2 pyhd8ed1ab_0 conda-forge tornado 6.1 py39h3811e60_2 conda-forge tqdm 4.63.1 pyhd8ed1ab_0 conda-forge traitlets 5.1.1 pyhd8ed1ab_0 conda-forge transformers 4.6.1 pyhd8ed1ab_0 conda-forge treelite 2.3.0 py39heaea588_0 conda-forge treelite-runtime 2.3.0 pypi_0 pypi twine 3.8.0 pyhd8ed1ab_0 conda-forge typed-ast 1.4.3 py39h3811e60_1 conda-forge typing-extensions 4.1.1 hd8ed1ab_0 conda-forge typing_extensions 4.1.1 pyha770c72_0 conda-forge tzcode 2022a h166bdaf_0 conda-forge tzdata 2022a h191b570_0 conda-forge ucx 1.12.0+gd367332 cuda11.2_0 rapidsai-nightly ucx-proc 1.0.0 gpu rapidsai-nightly ucx-py 0.25.00a220328 py39_gd367332_13 rapidsai-nightly ukkonen 1.0.1 py39h1a9c180_1 conda-forge umap-learn 0.5.2 py39hf3d152e_1 conda-forge unicodedata2 14.0.0 py39h3811e60_0 conda-forge urllib3 1.26.9 pyhd8ed1ab_0 conda-forge virtualenv 20.14.0 py39hf3d152e_0 conda-forge wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge webencodings 0.5.1 py_1 conda-forge websocket-client 1.3.1 pyhd8ed1ab_0 conda-forge websockets 10.2 py39hb9d737c_0 conda-forge werkzeug 2.0.3 pyhd8ed1ab_1 conda-forge wheel 0.37.1 pyhd8ed1ab_0 conda-forge widgetsnbextension 3.6.0 py39hf3d152e_0 conda-forge wrapt 1.14.0 py39hb9d737c_0 conda-forge xarray 2022.3.0 pyhd8ed1ab_0 conda-forge xerces-c 3.2.3 h9d8b166_3 conda-forge xgboost 1.5.2 pypi_0 pypi xmltodict 0.12.0 py_0 conda-forge xorg-kbproto 1.0.7 h7f98852_1002 conda-forge xorg-libice 1.0.10 h7f98852_0 conda-forge xorg-libsm 1.2.3 hd9c2040_1000 conda-forge xorg-libx11 1.7.2 h7f98852_0 conda-forge xorg-libxau 1.0.9 h7f98852_0 conda-forge xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge xorg-libxext 1.3.4 h7f98852_1 conda-forge xorg-libxrender 0.9.10 h7f98852_1003 conda-forge xorg-renderproto 0.11.1 h7f98852_1002 conda-forge xorg-xextproto 7.3.0 h7f98852_1002 conda-forge xorg-xproto 7.0.31 h7f98852_1007 conda-forge xz 5.2.5 h516909a_1 conda-forge yaml 0.2.5 h7f98852_2 conda-forge yarl 1.7.2 py39h3811e60_1 conda-forge zeromq 4.3.4 h9c3ff4c_1 conda-forge zfp 0.5.5 h9c3ff4c_8 conda-forge zict 2.1.0 pyhd8ed1ab_0 conda-forge zipp 3.7.0 pyhd8ed1ab_1 conda-forge zlib 1.2.11 h166bdaf_1014 conda-forge zstd 1.5.2 ha95c52a_0 conda-forge
This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.
I landed back on this issue while looking for UVM discussions in cuDF. How could we get an OOM error if UVM (Unified Virtual Memory) is enabled and working properly?
I've never seen an OOM with UVM, just full system hangs when host memory is exhausted.
Perhaps the original issue came out of UVM not working properly rather than a cuDF issues. @charlesbluca would you please share any updates since your original filing?
Sorry for the delay, went ahead and retried the script with updated nightlies and this still seems to be an issue:
Conda environment:
○ → conda list List of packages in environment: "/home/charlesb/micromamba/envs/cudf-23.10" Name Version Build Channel ───────────────────────────────────────────────────────────────────────────────────────────────────── _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge asttokens 2.2.1 pyhd8ed1ab_0 conda-forge aws-c-auth 0.7.0 hf8751d9_2 conda-forge aws-c-cal 0.6.0 h93469e0_0 conda-forge aws-c-common 0.8.23 hd590300_0 conda-forge aws-c-compression 0.2.17 h862ab75_1 conda-forge aws-c-event-stream 0.3.1 h9599702_1 conda-forge aws-c-http 0.7.11 hbe98c3e_0 conda-forge aws-c-io 0.13.28 h3870b5a_0 conda-forge aws-c-mqtt 0.8.14 h2e270ba_2 conda-forge aws-c-s3 0.3.13 heb0bb06_2 conda-forge aws-c-sdkutils 0.1.11 h862ab75_1 conda-forge aws-checksums 0.1.16 h862ab75_1 conda-forge aws-crt-cpp 0.20.3 he9c0e7f_4 conda-forge aws-sdk-cpp 1.10.57 hbc2ea52_17 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 pyhd8ed1ab_3 conda-forge backports.functools_lru_cache 1.6.5 pyhd8ed1ab_0 conda-forge bzip2 1.0.8 h7f98852_4 conda-forge c-ares 1.19.1 hd590300_0 conda-forge ca-certificates 2023.7.22 hbcca054_0 conda-forge cachetools 5.3.1 pyhd8ed1ab_0 conda-forge cubinlinker 0.3.0 py310hfdf336d_0 rapidsai-nightly cuda-python 11.8.2 py310h01a121a_0 conda-forge cuda-version 11.8 h70ddcb2_2 conda-forge cudatoolkit 11.8.0 h4ba93d1_12 conda-forge cudf 23.10.00a cuda11_py310_230807_ge92de8113d_60 rapidsai-nightly cupy 12.1.0 py310h53f8385_1 conda-forge decorator 5.1.1 pyhd8ed1ab_0 conda-forge dlpack 0.5 h9c3ff4c_0 conda-forge executing 1.2.0 pyhd8ed1ab_0 conda-forge fastrlock 0.8 py310hd8f1fbe_3 conda-forge fmt 9.1.0 h924138e_0 conda-forge fsspec 2023.6.0 pyh1a96a4e_0 conda-forge gflags 2.2.2 he1b5a44_1004 conda-forge glog 0.6.0 h6f12383_0 conda-forge gmock 1.14.0 ha770c72_0 conda-forge gtest 1.14.0 h00ab1b0_0 conda-forge ipython 8.14.0 pyh41d4057_0 conda-forge jedi 0.19.0 pyhd8ed1ab_0 conda-forge keyutils 1.6.1 h166bdaf_0 conda-forge krb5 1.21.1 h659d440_0 conda-forge ld_impl_linux-64 2.40 h41732ed_0 conda-forge libabseil 20230125.3 cxx17_h59595ed_0 conda-forge libarrow 12.0.1 h657c46f_7_cpu conda-forge libblas 3.9.0 17_linux64_openblas conda-forge libbrotlicommon 1.0.9 h166bdaf_9 conda-forge libbrotlidec 1.0.9 h166bdaf_9 conda-forge libbrotlienc 1.0.9 h166bdaf_9 conda-forge libcblas 3.9.0 17_linux64_openblas conda-forge libcrc32c 1.1.2 h9c3ff4c_0 conda-forge libcudf 23.10.00a cuda11_230807_ge92de8113d_60 rapidsai-nightly libcufile 1.4.0.31 0 nvidia libcufile-dev 1.4.0.31 0 nvidia libcurl 8.2.1 hca28451_0 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 h516909a_1 conda-forge libevent 2.1.12 hf998b51_1 conda-forge libffi 3.4.2 h7f98852_5 conda-forge libgcc-ng 13.1.0 he5830b7_0 conda-forge libgfortran-ng 13.1.0 h69a702a_0 conda-forge libgfortran5 13.1.0 h15d22d2_0 conda-forge libgomp 13.1.0 he5830b7_0 conda-forge libgoogle-cloud 2.12.0 h840a212_1 conda-forge libgrpc 1.56.2 h3905398_0 conda-forge libkvikio 23.10.00a cuda11_230807_g0247ca6_6 rapidsai-nightly liblapack 3.9.0 17_linux64_openblas conda-forge libllvm14 14.0.6 hcd5def8_4 conda-forge libnghttp2 1.52.0 h61bc06f_0 conda-forge libnsl 2.0.0 h7f98852_0 conda-forge libnuma 2.0.16 h0b41bf4_1 conda-forge libopenblas 0.3.23 pthreads_h80387f5_0 conda-forge libprotobuf 4.23.3 hd1fb520_0 conda-forge librmm 23.10.00a cuda11_230807_gcd37245e_9 rapidsai-nightly libsqlite 3.42.0 h2797004_0 conda-forge libssh2 1.11.0 h0841786_0 conda-forge libstdcxx-ng 13.1.0 hfd8a6a1_0 conda-forge libthrift 0.18.1 h8fd135c_2 conda-forge libutf8proc 2.8.0 h166bdaf_0 conda-forge libuuid 2.38.1 h0b41bf4_0 conda-forge libzlib 1.2.13 hd590300_5 conda-forge llvmlite 0.40.1 py310h1b8f574_0 conda-forge lz4-c 1.9.4 hcb278e6_0 conda-forge matplotlib-inline 0.1.6 pyhd8ed1ab_0 conda-forge ncurses 6.4 hcb278e6_0 conda-forge numba 0.57.1 py310h0f6aa51_0 conda-forge numpy 1.24.4 py310ha4c1d20_0 conda-forge nvcomp 2.6.1 h0800d71_2 conda-forge nvtx 0.2.5 py310h1fa729e_0 conda-forge openssl 3.1.2 hd590300_0 conda-forge orc 1.9.0 h385abfd_1 conda-forge packaging 23.1 pyhd8ed1ab_0 conda-forge pandas 1.5.3 py310h9b08913_1 conda-forge parso 0.8.3 pyhd8ed1ab_0 conda-forge pexpect 4.8.0 pyh1a96a4e_2 conda-forge pickleshare 0.7.5 py_1003 conda-forge pip 23.2.1 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.39 pyha770c72_0 conda-forge prompt_toolkit 3.0.39 hd8ed1ab_0 conda-forge protobuf 4.23.3 py310hb875b13_0 conda-forge ptxcompiler 0.8.1 py310h01a121a_0 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge pyarrow 12.0.1 py310h0576679_7_cpu conda-forge pygments 2.16.1 pyhd8ed1ab_0 conda-forge python 3.10.12 hd12c33a_0_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python_abi 3.10 3_cp310 conda-forge pytz 2023.3 pyhd8ed1ab_0 conda-forge rdma-core 28.9 h59595ed_1 conda-forge re2 2023.03.02 h8c504da_0 conda-forge readline 8.2 h8228510_1 conda-forge rmm 23.10.00a cuda11_py310_230807_gcd37245e_9 rapidsai-nightly s2n 1.3.46 h06160fa_0 conda-forge setuptools 68.0.0 pyhd8ed1ab_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge snappy 1.1.10 h9fff704_0 conda-forge spdlog 1.11.0 h9b3ece8_1 conda-forge stack_data 0.6.2 pyhd8ed1ab_0 conda-forge tk 8.6.12 h27826a3_0 conda-forge traitlets 5.9.0 pyhd8ed1ab_0 conda-forge typing_extensions 4.7.1 pyha770c72_0 conda-forge tzdata 2023c h71feb2d_0 conda-forge ucx 1.14.1 h4a2ce2d_2 conda-forge wcwidth 0.2.6 pyhd8ed1ab_0 conda-forge wheel 0.41.1 pyhd8ed1ab_0 conda-forge xz 5.2.6 h166bdaf_0 conda-forge zstd 1.5.2 hfc55251_7 conda-forge
Perhaps the original issue came out of UVM not working properly rather than a cuDF issues
Any additional information I could provide here to help narrow down the cause? 🙂
@charlesbluca I asked and it turns out that this is expected on WSL. The UVM support on windows display driver model (WDDM) is a limited form of UVM that doesn't support oversubscription (or simultaneous CPU / GPU access). Pages are not migrated on WDDM.
That said, there is a limited form of oversubscription that is supported for regular cudaMalloc calls. Could you do me a favor and try your script with this line commented out so that it uses the default memory resource?
rmm.mr.set_current_device_resource(rmm.mr.ManagedMemoryResource())
I kind of expect this to change or move the failure, rather than solve it, but it will be interesting to see.
Tested the script below on my WSL2 (Win 11/Ubuntu22.04) machine that has an RTX4090:
import cupy as cp
import cudf
import os
def generate_random_data(num_rows, num_columns):
"""Generate random numbers using CuPy and return a cuDF DataFrame."""
# Generate a random CuPy array
data = cp.random.rand(num_rows, num_columns)
# Convert to cuDF DataFrame
df = cudf.DataFrame(data, columns=[f'col_{i}' for i in range(num_columns)])
return df
def generate_csv_gpu(target_size, filename='cudf_uvm_data_20gb.csv'):
num_columns = 5
row_estimate = 1000 # Initial guess for number of rows
# Generate initial data
df = generate_random_data(row_estimate, num_columns)
df.to_csv(filename, index=False)
# Check file size and adjust
current_size = os.path.getsize(filename)
row_size = current_size / row_estimate
total_rows_needed = int(target_size / row_size)
# Generate the correct amount of data
df = generate_random_data(total_rows_needed, num_columns)
df.to_csv(filename, index=False, chunksize=1000000)
# Report final file size
final_size = os.path.getsize(filename)
print(f"Targeted file size was {target_size} bytes.")
print(f"Final file size is {final_size} bytes.")
# Usage example
generate_csv_gpu(20000000000)
input_data_path ='cudf_uvm_data_20gb.csv'
df = cudf.read_csv(input_data_path)
len(df)
>>> 336309673
So no issues with a CSV that's ~18gb with an available 24gb GPU memory. In the Windows task manager GPU perf tab we can see it needs to jump into the shared memory pool, but does so successfully.
So - I think this has been resolved at some point in WSL.
Confirmed internally that this means it is over subscribing. OK so the behavior with managed memory is expected, but this option is available.
I think we can close this now.