storage
storage copied to clipboard
Published
20 hours ago •
mlcommons
mlcommons
need help on FedoraOS 37 installation error
hello team, I install depends follow the guide, but I suffer this on FedoraOS37
(base) [root@cassiopeia storage]# pip3 install -r dlio_benchmark/requirements.txt
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cpu, https://developer.download.nvidia.com/compute/redist
Collecting absl-py==1.3.0 (from -r dlio_benchmark/requirements.txt (line 5))
Using cached absl_py-1.3.0-py3-none-any.whl.metadata (2.3 kB)
Collecting antlr4-python3-runtime==4.9.3 (from -r dlio_benchmark/requirements.txt (line 6))
Using cached antlr4-python3-runtime-4.9.3.tar.gz (117 kB)
Preparing metadata (setup.py) ... done
Collecting astunparse==1.6.3 (from -r dlio_benchmark/requirements.txt (line 7))
Using cached astunparse-1.6.3-py2.py3-none-any.whl.metadata (4.4 kB)
Collecting cachetools==5.2.0 (from -r dlio_benchmark/requirements.txt (line 8))
Using cached cachetools-5.2.0-py3-none-any.whl.metadata (5.1 kB)
Collecting certifi==2022.9.24 (from -r dlio_benchmark/requirements.txt (line 9))
Using cached certifi-2022.9.24-py3-none-any.whl.metadata (2.9 kB)
Collecting charset-normalizer==2.1.1 (from -r dlio_benchmark/requirements.txt (line 10))
Using cached https://download.pytorch.org/whl/charset_normalizer-2.1.1-py3-none-any.whl (39 kB)
Collecting dlio_profiler_py==0.0.3 (from -r dlio_benchmark/requirements.txt (line 11))
Using cached dlio_profiler_py-0.0.3.tar.gz (41 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting flatbuffers==23.5.26 (from -r dlio_benchmark/requirements.txt (line 12))
Using cached flatbuffers-23.5.26-py2.py3-none-any.whl.metadata (850 bytes)
Collecting gast==0.4.0 (from -r dlio_benchmark/requirements.txt (line 13))
Using cached gast-0.4.0-py3-none-any.whl.metadata (1.1 kB)
Collecting google-auth==2.14.1 (from -r dlio_benchmark/requirements.txt (line 14))
Using cached google_auth-2.14.1-py2.py3-none-any.whl.metadata (4.2 kB)
Collecting google-auth-oauthlib==0.7.0 (from -r dlio_benchmark/requirements.txt (line 15))
Using cached google_auth_oauthlib-0.7.0-py2.py3-none-any.whl.metadata (2.6 kB)
Collecting google-pasta==0.2.0 (from -r dlio_benchmark/requirements.txt (line 16))
Using cached google_pasta-0.2.0-py3-none-any.whl.metadata (814 bytes)
Collecting grpcio==1.51.0 (from -r dlio_benchmark/requirements.txt (line 17))
Using cached grpcio-1.51.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.8 kB)
Collecting h5py==3.7.0 (from -r dlio_benchmark/requirements.txt (line 18))
Using cached h5py-3.7.0.tar.gz (392 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting hydra-core==1.2.0 (from -r dlio_benchmark/requirements.txt (line 19))
Using cached hydra_core-1.2.0-py3-none-any.whl.metadata (4.0 kB)
Collecting idna==3.4 (from -r dlio_benchmark/requirements.txt (line 20))
Using cached https://download.pytorch.org/whl/idna-3.4-py3-none-any.whl (61 kB)
Collecting keras==2.15.0 (from -r dlio_benchmark/requirements.txt (line 21))
Using cached keras-2.15.0-py3-none-any.whl.metadata (2.4 kB)
Collecting libclang==14.0.6 (from -r dlio_benchmark/requirements.txt (line 22))
Using cached libclang-14.0.6-py2.py3-none-manylinux2010_x86_64.whl.metadata (5.0 kB)
Collecting Markdown==3.4.1 (from -r dlio_benchmark/requirements.txt (line 23))
Using cached Markdown-3.4.1-py3-none-any.whl.metadata (4.6 kB)
Collecting MarkupSafe==2.1.1 (from -r dlio_benchmark/requirements.txt (line 24))
Using cached MarkupSafe-2.1.1.tar.gz (18 kB)
Preparing metadata (setup.py) ... done
Collecting mpi4py==3.1.4 (from -r dlio_benchmark/requirements.txt (line 25))
Using cached mpi4py-3.1.4.tar.gz (2.5 MB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting numpy==1.23.5 (from -r dlio_benchmark/requirements.txt (line 26))
Using cached numpy-1.23.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.3 kB)
Collecting nvidia-dali-cuda110==1.34.0 (from -r dlio_benchmark/requirements.txt (line 27))
Using cached https://developer.download.nvidia.com/compute/redist/nvidia-dali-cuda110/nvidia_dali_cuda110-1.34.0-12152783-py3-none-manylinux2014_x86_64.whl (502.2 MB)
Collecting oauthlib==3.2.2 (from -r dlio_benchmark/requirements.txt (line 28))
Using cached oauthlib-3.2.2-py3-none-any.whl.metadata (7.5 kB)
Collecting omegaconf==2.2.3 (from -r dlio_benchmark/requirements.txt (line 29))
Using cached omegaconf-2.2.3-py3-none-any.whl.metadata (3.9 kB)
Collecting opt-einsum==3.3.0 (from -r dlio_benchmark/requirements.txt (line 30))
Using cached opt_einsum-3.3.0-py3-none-any.whl.metadata (6.5 kB)
Collecting packaging==21.3 (from -r dlio_benchmark/requirements.txt (line 31))
Using cached packaging-21.3-py3-none-any.whl.metadata (15 kB)
Collecting pandas==1.5.1 (from -r dlio_benchmark/requirements.txt (line 32))
Using cached pandas-1.5.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (11 kB)
Collecting Pillow==9.3.0 (from -r dlio_benchmark/requirements.txt (line 33))
Using cached Pillow-9.3.0-cp311-cp311-manylinux_2_28_x86_64.whl.metadata (9.2 kB)
Collecting protobuf==4.23.4 (from -r dlio_benchmark/requirements.txt (line 34))
Using cached protobuf-4.23.4-cp37-abi3-manylinux2014_x86_64.whl.metadata (540 bytes)
Collecting psutil==5.9.8 (from -r dlio_benchmark/requirements.txt (line 35))
Using cached psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (21 kB)
Collecting pyasn1==0.4.8 (from -r dlio_benchmark/requirements.txt (line 36))
Using cached pyasn1-0.4.8-py2.py3-none-any.whl.metadata (1.5 kB)
Collecting pyasn1-modules==0.2.8 (from -r dlio_benchmark/requirements.txt (line 37))
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl.metadata (1.9 kB)
Requirement already satisfied: pyparsing==3.0.9 in /usr/local/lib/python3.11/site-packages (from -r dlio_benchmark/requirements.txt (line 38)) (3.0.9)
Requirement already satisfied: python-dateutil==2.8.2 in /usr/local/lib/python3.11/site-packages (from -r dlio_benchmark/requirements.txt (line 39)) (2.8.2)
Collecting pytz==2022.6 (from -r dlio_benchmark/requirements.txt (line 40))
Using cached pytz-2022.6-py2.py3-none-any.whl.metadata (21 kB)
Requirement already satisfied: PyYAML==6.0 in /usr/local/lib64/python3.11/site-packages (from -r dlio_benchmark/requirements.txt (line 41)) (6.0)
Collecting requests==2.28.1 (from -r dlio_benchmark/requirements.txt (line 42))
Using cached https://download.pytorch.org/whl/requests-2.28.1-py3-none-any.whl (62 kB)
Collecting requests-oauthlib==1.3.1 (from -r dlio_benchmark/requirements.txt (line 43))
Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl.metadata (10 kB)
Collecting rsa==4.9 (from -r dlio_benchmark/requirements.txt (line 44))
Using cached rsa-4.9-py3-none-any.whl.metadata (4.2 kB)
Requirement already satisfied: six==1.16.0 in /usr/local/lib/python3.11/site-packages (from -r dlio_benchmark/requirements.txt (line 45)) (1.16.0)
Collecting tensorboard==2.15.0 (from -r dlio_benchmark/requirements.txt (line 46))
Using cached tensorboard-2.15.0-py3-none-any.whl.metadata (1.7 kB)
Collecting tensorboard-data-server==0.7.2 (from -r dlio_benchmark/requirements.txt (line 47))
Using cached tensorboard_data_server-0.7.2-py3-none-manylinux_2_31_x86_64.whl.metadata (1.1 kB)
Collecting tensorboard-plugin-wit==1.8.1 (from -r dlio_benchmark/requirements.txt (line 48))
Using cached tensorboard_plugin_wit-1.8.1-py3-none-any.whl.metadata (873 bytes)
Collecting tensorflow-cpu==2.15.0 (from -r dlio_benchmark/requirements.txt (line 49))
Using cached tensorflow_cpu-2.15.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.4 kB)
ERROR: Ignored the following versions that require a different python version: 0.28.0 Requires-Python >=3.7, <3.11; 1.16.0 Requires-Python >=3.6, <3.11; 1.21.2 Requires-Python >=3.7,<3.11; 1.21.3 Requires-Python >=3.7,<3.11; 1.21.4 Requires-Python >=3.7,<3.11; 1.21.5 Requires-Python >=3.7,<3.11; 1.21.6 Requires-Python >=3.7,<3.11
ERROR: Could not find a version that satisfies the requirement tensorflow-io==0.28.0 (from versions: 0.29.0, 0.30.0, 0.31.0, 0.32.0, 0.33.0, 0.34.0, 0.35.0, 0.36.0)
ERROR: No matching distribution found for tensorflow-io==0.28.0
guys, I found this guide runs well on ubuntu 22.04 server but not FedoraOS37, could you please fix this?
@gaowayne , it is due to the python version you used in FedoraOS37. For a simple fix, please try modifying the requirements.txt, change to tensorflow-io==0.29.0
@gaowayne , it is due to the python version you used in FedoraOS37. For a simple fix, please try modifying the requirements.txt, change to tensorflow-io==0.29.0
this does help. now Fedora installation is done. but after I run below command, it still failed. ubuntu server works fine for all things.
(base) [root@cassiopeia storage]# ./benchmark.sh datagen --workload unet3d --accelerator-type h100 --num-parallel 8 --param dataset.num_files_train=1200 --param dataset.data_folder=/mnt/nvme6n1/mlperf/
hwloc/linux: Ignoring PCI device with non-16bit domain.
Pass --enable-32bits-pci-domain to configure to support such devices
(warning: it would break the library ABI, don't enable unless really needed).
Traceback (most recent call last):
File "/mnt/nvme1n1/mlperf/storage/dlio_benchmark/dlio_benchmark/main.py", line 19, in <module>
import hydra
File "/usr/local/lib/python3.11/site-packages/hydra/__init__.py", line 5, in <module>
from hydra import utils
File "/usr/local/lib/python3.11/site-packages/hydra/utils.py", line 8, in <module>
import hydra._internal.instantiate._instantiate2
File "/usr/local/lib/python3.11/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 12, in <module>
from hydra._internal.utils import _locate
File "/usr/local/lib/python3.11/site-packages/hydra/_internal/utils.py", line 18, in <module>
from hydra.core.utils import get_valid_filename, validate_config_path
File "/usr/local/lib/python3.11/site-packages/hydra/core/utils.py", line 20, in <module>
from hydra.core.hydra_config import HydraConfig
File "/usr/local/lib/python3.11/site-packages/hydra/core/hydra_config.py", line 6, in <module>
from hydra.conf import HydraConf
File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 46, in <module>
class JobConf:
File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 75, in JobConf
@dataclass
^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 1230, in dataclass
return wrap(cls)
^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 1220, in wrap
return _process_class(cls, init, repr, eq, order, unsafe_hash,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 958, in _process_class
cls_fields.append(_get_field(cls, name, type, kw_only))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 815, in _get_field
raise ValueError(f'mutable default {type(f.default)} for field '
ValueError: mutable default <class 'hydra.conf.JobConf.JobConfig.OverrideDirname'> for field override_dirname is not allowed: use default_factory
Traceback (most recent call last):
File "/mnt/nvme1n1/mlperf/storage/dlio_benchmark/dlio_benchmark/main.py", line 19, in <module>
Traceback (most recent call last):
File "/mnt/nvme1n1/mlperf/storage/dlio_benchmark/dlio_benchmark/main.py", line 19, in <module>
import hydra
File "/usr/local/lib/python3.11/site-packages/hydra/__init__.py", line 5, in <module>
import hydra
File "/usr/local/lib/python3.11/site-packages/hydra/__init__.py", line 5, in <module>
from hydra import utils
File "/usr/local/lib/python3.11/site-packages/hydra/utils.py", line 8, in <module>
from hydra import utils
import hydra._internal.instantiate._instantiate2
File "/usr/local/lib/python3.11/site-packages/hydra/utils.py", line 8, in <module>
File "/usr/local/lib/python3.11/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 12, in <module>
from hydra._internal.utils import _locate
File "/usr/local/lib/python3.11/site-packages/hydra/_internal/utils.py", line 18, in <module>
import hydra._internal.instantiate._instantiate2
File "/usr/local/lib/python3.11/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 12, in <module>
from hydra.core.utils import get_valid_filename, validate_config_path
File "/usr/local/lib/python3.11/site-packages/hydra/core/utils.py", line 20, in <module>
from hydra._internal.utils import _locate
File "/usr/local/lib/python3.11/site-packages/hydra/_internal/utils.py", line 18, in <module>
from hydra.core.hydra_config import HydraConfig
File "/usr/local/lib/python3.11/site-packages/hydra/core/hydra_config.py", line 6, in <module>
from hydra.core.utils import get_valid_filename, validate_config_path
File "/usr/local/lib/python3.11/site-packages/hydra/core/utils.py", line 20, in <module>
from hydra.conf import HydraConf
File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 46, in <module>
from hydra.core.hydra_config import HydraConfig
File "/usr/local/lib/python3.11/site-packages/hydra/core/hydra_config.py", line 6, in <module>
class JobConf:
File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 75, in JobConf
from hydra.conf import HydraConf
File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 46, in <module>
@dataclass
^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 1230, in dataclass
class JobConf:
File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 75, in JobConf
@dataclass
^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 1230, in dataclass
return wrap(cls)
^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 1220, in wrap
return wrap(cls)
^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 1220, in wrap
return _process_class(cls, init, repr, eq, order, unsafe_hash,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 958, in _process_class
return _process_class(cls, init, repr, eq, order, unsafe_hash,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 958, in _process_class
cls_fields.append(_get_field(cls, name, type, kw_only))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 815, in _get_field
cls_fields.append(_get_field(cls, name, type, kw_only))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 815, in _get_field
raise ValueError(f'mutable default {type(f.default)} for field '
ValueError: mutable default <class 'hydra.conf.JobConf.JobConfig.OverrideDirname'> for field override_dirname is not allowed: use default_factory
raise ValueError(f'mutable default {type(f.default)} for field '
ValueError: mutable default <class 'hydra.conf.JobConf.JobConfig.OverrideDirname'> for field override_dirname is not allowed: use default_factory
Traceback (most recent call last):
File "/mnt/nvme1n1/mlperf/storage/dlio_benchmark/dlio_benchmark/main.py", line 19, in <module>
import hydra
File "/usr/local/lib/python3.11/site-packages/hydra/__init__.py", line 5, in <module>
from hydra import utils
File "/usr/local/lib/python3.11/site-packages/hydra/utils.py", line 8, in <module>
import hydra._internal.instantiate._instantiate2
File "/usr/local/lib/python3.11/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 12, in <module>
from hydra._internal.utils import _locate
File "/usr/local/lib/python3.11/site-packages/hydra/_internal/utils.py", line 18, in <module>
from hydra.core.utils import get_valid_filename, validate_config_path
File "/usr/local/lib/python3.11/site-packages/hydra/core/utils.py", line 20, in <module>
from hydra.core.hydra_config import HydraConfig
File "/usr/local/lib/python3.11/site-packages/hydra/core/hydra_config.py", line 6, in <module>
from hydra.conf import HydraConf
File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 46, in <module>
class JobConf:
File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 75, in JobConf
@dataclass
^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 1230, in dataclass
return wrap(cls)
^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 1220, in wrap
Traceback (most recent call last):
File "/mnt/nvme1n1/mlperf/storage/dlio_benchmark/dlio_benchmark/main.py", line 19, in <module>
return _process_class(cls, init, repr, eq, order, unsafe_hash,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 958, in _process_class
cls_fields.append(_get_field(cls, name, type, kw_only))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 815, in _get_field
import hydra
File "/usr/local/lib/python3.11/site-packages/hydra/__init__.py", line 5, in <module>
from hydra import utils
File "/usr/local/lib/python3.11/site-packages/hydra/utils.py", line 8, in <module>
import hydra._internal.instantiate._instantiate2
File "/usr/local/lib/python3.11/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 12, in <module>
from hydra._internal.utils import _locate
File "/usr/local/lib/python3.11/site-packages/hydra/_internal/utils.py", line 18, in <module>
from hydra.core.utils import get_valid_filename, validate_config_path
File "/usr/local/lib/python3.11/site-packages/hydra/core/utils.py", line 20, in <module>
raise ValueError(f'mutable default {type(f.default)} for field '
from hydra.core.hydra_config import HydraConfig
File "/usr/local/lib/python3.11/site-packages/hydra/core/hydra_config.py", line 6, in <module>
from hydra.conf import HydraConf
ValueError: mutable default <class 'hydra.conf.JobConf.JobConfig.OverrideDirname'> for field override_dirname is not allowed: use default_factory
File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 46, in <module>
class JobConf:
File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 75, in JobConf
@dataclass
^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 1230, in dataclass
return wrap(cls)
^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 1220, in wrap
return _process_class(cls, init, repr, eq, order, unsafe_hash,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 958, in _process_class
cls_fields.append(_get_field(cls, name, type, kw_only))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 815, in _get_field
raise ValueError(f'mutable default {type(f.default)} for field '
ValueError: mutable default <class 'hydra.conf.JobConf.JobConfig.OverrideDirname'> for field override_dirname is not allowed: use default_factory
Traceback (most recent call last):
File "/mnt/nvme1n1/mlperf/storage/dlio_benchmark/dlio_benchmark/main.py", line 19, in <module>
import hydra
File "/usr/local/lib/python3.11/site-packages/hydra/__init__.py", line 5, in <module>
from hydra import utils
File "/usr/local/lib/python3.11/site-packages/hydra/utils.py", line 8, in <module>
import hydra._internal.instantiate._instantiate2
File "/usr/local/lib/python3.11/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 12, in <module>
from hydra._internal.utils import _locate
File "/usr/local/lib/python3.11/site-packages/hydra/_internal/utils.py", line 18, in <module>
from hydra.core.utils import get_valid_filename, validate_config_path
File "/usr/local/lib/python3.11/site-packages/hydra/core/utils.py", line 20, in <module>
from hydra.core.hydra_config import HydraConfig
File "/usr/local/lib/python3.11/site-packages/hydra/core/hydra_config.py", line 6, in <module>
from hydra.conf import HydraConf
File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 46, in <module>
class JobConf:
File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 75, in JobConf
@dataclass
^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 1230, in dataclass
return wrap(cls)
^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 1220, in wrap
return _process_class(cls, init, repr, eq, order, unsafe_hash,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 958, in _process_class
cls_fields.append(_get_field(cls, name, type, kw_only))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 815, in _get_field
raise ValueError(f'mutable default {type(f.default)} for field '
ValueError: mutable default <class 'hydra.conf.JobConf.JobConfig.OverrideDirname'> for field override_dirname is not allowed: use default_factory
Traceback (most recent call last):
File "/mnt/nvme1n1/mlperf/storage/dlio_benchmark/dlio_benchmark/main.py", line 19, in <module>
import hydra
File "/usr/local/lib/python3.11/site-packages/hydra/__init__.py", line 5, in <module>
Traceback (most recent call last):
File "/mnt/nvme1n1/mlperf/storage/dlio_benchmark/dlio_benchmark/main.py", line 19, in <module>
from hydra import utils
File "/usr/local/lib/python3.11/site-packages/hydra/utils.py", line 8, in <module>
import hydra._internal.instantiate._instantiate2
File "/usr/local/lib/python3.11/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 12, in <module>
from hydra._internal.utils import _locate
File "/usr/local/lib/python3.11/site-packages/hydra/_internal/utils.py", line 18, in <module>
from hydra.core.utils import get_valid_filename, validate_config_path
File "/usr/local/lib/python3.11/site-packages/hydra/core/utils.py", line 20, in <module>
from hydra.core.hydra_config import HydraConfig
File "/usr/local/lib/python3.11/site-packages/hydra/core/hydra_config.py", line 6, in <module>
from hydra.conf import HydraConf
File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 46, in <module>
class JobConf:
File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 75, in JobConf
@dataclass
^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 1230, in dataclass
import hydra
File "/usr/local/lib/python3.11/site-packages/hydra/__init__.py", line 5, in <module>
from hydra import utils
File "/usr/local/lib/python3.11/site-packages/hydra/utils.py", line 8, in <module>
import hydra._internal.instantiate._instantiate2
File "/usr/local/lib/python3.11/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 12, in <module>
from hydra._internal.utils import _locate
File "/usr/local/lib/python3.11/site-packages/hydra/_internal/utils.py", line 18, in <module>
return wrap(cls)
^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 1220, in wrap
from hydra.core.utils import get_valid_filename, validate_config_path
File "/usr/local/lib/python3.11/site-packages/hydra/core/utils.py", line 20, in <module>
from hydra.core.hydra_config import HydraConfig
File "/usr/local/lib/python3.11/site-packages/hydra/core/hydra_config.py", line 6, in <module>
from hydra.conf import HydraConf
File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 46, in <module>
class JobConf:
File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 75, in JobConf
@dataclass
^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 1230, in dataclass
return _process_class(cls, init, repr, eq, order, unsafe_hash,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 958, in _process_class
return wrap(cls)
^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 1220, in wrap
cls_fields.append(_get_field(cls, name, type, kw_only))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 815, in _get_field
return _process_class(cls, init, repr, eq, order, unsafe_hash,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 958, in _process_class
raise ValueError(f'mutable default {type(f.default)} for field '
ValueError: mutable default <class 'hydra.conf.JobConf.JobConfig.OverrideDirname'> for field override_dirname is not allowed: use default_factory
cls_fields.append(_get_field(cls, name, type, kw_only))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 815, in _get_field
raise ValueError(f'mutable default {type(f.default)} for field '
ValueError: mutable default <class 'hydra.conf.JobConf.JobConfig.OverrideDirname'> for field override_dirname is not allowed: use default_factory
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[32296,1],1]
Exit code: 1
--------------------------------------------------------------------------
(base) [root@cassiopeia storage]#
@gaowayne , it is due to the python version you used in FedoraOS37. For a simple fix, please try modifying the requirements.txt, change to tensorflow-io==0.29.0
this does help. now Fedora installation is done. but after I run below command, it still failed. ubuntu server works fine for all things.
(base) [root@cassiopeia storage]# ./benchmark.sh datagen --workload unet3d --accelerator-type h100 --num-parallel 8 --param dataset.num_files_train=1200 --param dataset.data_folder=/mnt/nvme6n1/mlperf/ hwloc/linux: Ignoring PCI device with non-16bit domain. Pass --enable-32bits-pci-domain to configure to support such devices (warning: it would break the library ABI, don't enable unless really needed). Traceback (most recent call last): File "/mnt/nvme1n1/mlperf/storage/dlio_benchmark/dlio_benchmark/main.py", line 19, in <module> import hydra File "/usr/local/lib/python3.11/site-packages/hydra/__init__.py", line 5, in <module> from hydra import utils File "/usr/local/lib/python3.11/site-packages/hydra/utils.py", line 8, in <module> import hydra._internal.instantiate._instantiate2 File "/usr/local/lib/python3.11/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 12, in <module> from hydra._internal.utils import _locate File "/usr/local/lib/python3.11/site-packages/hydra/_internal/utils.py", line 18, in <module> from hydra.core.utils import get_valid_filename, validate_config_path File "/usr/local/lib/python3.11/site-packages/hydra/core/utils.py", line 20, in <module> from hydra.core.hydra_config import HydraConfig File "/usr/local/lib/python3.11/site-packages/hydra/core/hydra_config.py", line 6, in <module> from hydra.conf import HydraConf File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 46, in <module> class JobConf: File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 75, in JobConf @dataclass ^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 1230, in dataclass return wrap(cls) ^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 1220, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 958, in _process_class cls_fields.append(_get_field(cls, name, type, kw_only)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 815, in _get_field raise ValueError(f'mutable default {type(f.default)} for field ' ValueError: mutable default <class 'hydra.conf.JobConf.JobConfig.OverrideDirname'> for field override_dirname is not allowed: use default_factory Traceback (most recent call last): File "/mnt/nvme1n1/mlperf/storage/dlio_benchmark/dlio_benchmark/main.py", line 19, in <module> Traceback (most recent call last): File "/mnt/nvme1n1/mlperf/storage/dlio_benchmark/dlio_benchmark/main.py", line 19, in <module> import hydra File "/usr/local/lib/python3.11/site-packages/hydra/__init__.py", line 5, in <module> import hydra File "/usr/local/lib/python3.11/site-packages/hydra/__init__.py", line 5, in <module> from hydra import utils File "/usr/local/lib/python3.11/site-packages/hydra/utils.py", line 8, in <module> from hydra import utils import hydra._internal.instantiate._instantiate2 File "/usr/local/lib/python3.11/site-packages/hydra/utils.py", line 8, in <module> File "/usr/local/lib/python3.11/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 12, in <module> from hydra._internal.utils import _locate File "/usr/local/lib/python3.11/site-packages/hydra/_internal/utils.py", line 18, in <module> import hydra._internal.instantiate._instantiate2 File "/usr/local/lib/python3.11/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 12, in <module> from hydra.core.utils import get_valid_filename, validate_config_path File "/usr/local/lib/python3.11/site-packages/hydra/core/utils.py", line 20, in <module> from hydra._internal.utils import _locate File "/usr/local/lib/python3.11/site-packages/hydra/_internal/utils.py", line 18, in <module> from hydra.core.hydra_config import HydraConfig File "/usr/local/lib/python3.11/site-packages/hydra/core/hydra_config.py", line 6, in <module> from hydra.core.utils import get_valid_filename, validate_config_path File "/usr/local/lib/python3.11/site-packages/hydra/core/utils.py", line 20, in <module> from hydra.conf import HydraConf File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 46, in <module> from hydra.core.hydra_config import HydraConfig File "/usr/local/lib/python3.11/site-packages/hydra/core/hydra_config.py", line 6, in <module> class JobConf: File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 75, in JobConf from hydra.conf import HydraConf File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 46, in <module> @dataclass ^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 1230, in dataclass class JobConf: File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 75, in JobConf @dataclass ^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 1230, in dataclass return wrap(cls) ^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 1220, in wrap return wrap(cls) ^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 1220, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 958, in _process_class return _process_class(cls, init, repr, eq, order, unsafe_hash, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 958, in _process_class cls_fields.append(_get_field(cls, name, type, kw_only)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 815, in _get_field cls_fields.append(_get_field(cls, name, type, kw_only)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 815, in _get_field raise ValueError(f'mutable default {type(f.default)} for field ' ValueError: mutable default <class 'hydra.conf.JobConf.JobConfig.OverrideDirname'> for field override_dirname is not allowed: use default_factory raise ValueError(f'mutable default {type(f.default)} for field ' ValueError: mutable default <class 'hydra.conf.JobConf.JobConfig.OverrideDirname'> for field override_dirname is not allowed: use default_factory Traceback (most recent call last): File "/mnt/nvme1n1/mlperf/storage/dlio_benchmark/dlio_benchmark/main.py", line 19, in <module> import hydra File "/usr/local/lib/python3.11/site-packages/hydra/__init__.py", line 5, in <module> from hydra import utils File "/usr/local/lib/python3.11/site-packages/hydra/utils.py", line 8, in <module> import hydra._internal.instantiate._instantiate2 File "/usr/local/lib/python3.11/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 12, in <module> from hydra._internal.utils import _locate File "/usr/local/lib/python3.11/site-packages/hydra/_internal/utils.py", line 18, in <module> from hydra.core.utils import get_valid_filename, validate_config_path File "/usr/local/lib/python3.11/site-packages/hydra/core/utils.py", line 20, in <module> from hydra.core.hydra_config import HydraConfig File "/usr/local/lib/python3.11/site-packages/hydra/core/hydra_config.py", line 6, in <module> from hydra.conf import HydraConf File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 46, in <module> class JobConf: File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 75, in JobConf @dataclass ^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 1230, in dataclass return wrap(cls) ^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 1220, in wrap Traceback (most recent call last): File "/mnt/nvme1n1/mlperf/storage/dlio_benchmark/dlio_benchmark/main.py", line 19, in <module> return _process_class(cls, init, repr, eq, order, unsafe_hash, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 958, in _process_class cls_fields.append(_get_field(cls, name, type, kw_only)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 815, in _get_field import hydra File "/usr/local/lib/python3.11/site-packages/hydra/__init__.py", line 5, in <module> from hydra import utils File "/usr/local/lib/python3.11/site-packages/hydra/utils.py", line 8, in <module> import hydra._internal.instantiate._instantiate2 File "/usr/local/lib/python3.11/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 12, in <module> from hydra._internal.utils import _locate File "/usr/local/lib/python3.11/site-packages/hydra/_internal/utils.py", line 18, in <module> from hydra.core.utils import get_valid_filename, validate_config_path File "/usr/local/lib/python3.11/site-packages/hydra/core/utils.py", line 20, in <module> raise ValueError(f'mutable default {type(f.default)} for field ' from hydra.core.hydra_config import HydraConfig File "/usr/local/lib/python3.11/site-packages/hydra/core/hydra_config.py", line 6, in <module> from hydra.conf import HydraConf ValueError: mutable default <class 'hydra.conf.JobConf.JobConfig.OverrideDirname'> for field override_dirname is not allowed: use default_factory File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 46, in <module> class JobConf: File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 75, in JobConf @dataclass ^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 1230, in dataclass return wrap(cls) ^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 1220, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 958, in _process_class cls_fields.append(_get_field(cls, name, type, kw_only)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 815, in _get_field raise ValueError(f'mutable default {type(f.default)} for field ' ValueError: mutable default <class 'hydra.conf.JobConf.JobConfig.OverrideDirname'> for field override_dirname is not allowed: use default_factory Traceback (most recent call last): File "/mnt/nvme1n1/mlperf/storage/dlio_benchmark/dlio_benchmark/main.py", line 19, in <module> import hydra File "/usr/local/lib/python3.11/site-packages/hydra/__init__.py", line 5, in <module> from hydra import utils File "/usr/local/lib/python3.11/site-packages/hydra/utils.py", line 8, in <module> import hydra._internal.instantiate._instantiate2 File "/usr/local/lib/python3.11/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 12, in <module> from hydra._internal.utils import _locate File "/usr/local/lib/python3.11/site-packages/hydra/_internal/utils.py", line 18, in <module> from hydra.core.utils import get_valid_filename, validate_config_path File "/usr/local/lib/python3.11/site-packages/hydra/core/utils.py", line 20, in <module> from hydra.core.hydra_config import HydraConfig File "/usr/local/lib/python3.11/site-packages/hydra/core/hydra_config.py", line 6, in <module> from hydra.conf import HydraConf File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 46, in <module> class JobConf: File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 75, in JobConf @dataclass ^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 1230, in dataclass return wrap(cls) ^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 1220, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 958, in _process_class cls_fields.append(_get_field(cls, name, type, kw_only)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 815, in _get_field raise ValueError(f'mutable default {type(f.default)} for field ' ValueError: mutable default <class 'hydra.conf.JobConf.JobConfig.OverrideDirname'> for field override_dirname is not allowed: use default_factory Traceback (most recent call last): File "/mnt/nvme1n1/mlperf/storage/dlio_benchmark/dlio_benchmark/main.py", line 19, in <module> import hydra File "/usr/local/lib/python3.11/site-packages/hydra/__init__.py", line 5, in <module> Traceback (most recent call last): File "/mnt/nvme1n1/mlperf/storage/dlio_benchmark/dlio_benchmark/main.py", line 19, in <module> from hydra import utils File "/usr/local/lib/python3.11/site-packages/hydra/utils.py", line 8, in <module> import hydra._internal.instantiate._instantiate2 File "/usr/local/lib/python3.11/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 12, in <module> from hydra._internal.utils import _locate File "/usr/local/lib/python3.11/site-packages/hydra/_internal/utils.py", line 18, in <module> from hydra.core.utils import get_valid_filename, validate_config_path File "/usr/local/lib/python3.11/site-packages/hydra/core/utils.py", line 20, in <module> from hydra.core.hydra_config import HydraConfig File "/usr/local/lib/python3.11/site-packages/hydra/core/hydra_config.py", line 6, in <module> from hydra.conf import HydraConf File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 46, in <module> class JobConf: File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 75, in JobConf @dataclass ^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 1230, in dataclass import hydra File "/usr/local/lib/python3.11/site-packages/hydra/__init__.py", line 5, in <module> from hydra import utils File "/usr/local/lib/python3.11/site-packages/hydra/utils.py", line 8, in <module> import hydra._internal.instantiate._instantiate2 File "/usr/local/lib/python3.11/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 12, in <module> from hydra._internal.utils import _locate File "/usr/local/lib/python3.11/site-packages/hydra/_internal/utils.py", line 18, in <module> return wrap(cls) ^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 1220, in wrap from hydra.core.utils import get_valid_filename, validate_config_path File "/usr/local/lib/python3.11/site-packages/hydra/core/utils.py", line 20, in <module> from hydra.core.hydra_config import HydraConfig File "/usr/local/lib/python3.11/site-packages/hydra/core/hydra_config.py", line 6, in <module> from hydra.conf import HydraConf File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 46, in <module> class JobConf: File "/usr/local/lib/python3.11/site-packages/hydra/conf/__init__.py", line 75, in JobConf @dataclass ^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 1230, in dataclass return _process_class(cls, init, repr, eq, order, unsafe_hash, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 958, in _process_class return wrap(cls) ^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 1220, in wrap cls_fields.append(_get_field(cls, name, type, kw_only)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 815, in _get_field return _process_class(cls, init, repr, eq, order, unsafe_hash, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 958, in _process_class raise ValueError(f'mutable default {type(f.default)} for field ' ValueError: mutable default <class 'hydra.conf.JobConf.JobConfig.OverrideDirname'> for field override_dirname is not allowed: use default_factory cls_fields.append(_get_field(cls, name, type, kw_only)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/dataclasses.py", line 815, in _get_field raise ValueError(f'mutable default {type(f.default)} for field ' ValueError: mutable default <class 'hydra.conf.JobConf.JobConfig.OverrideDirname'> for field override_dirname is not allowed: use default_factory -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- -------------------------------------------------------------------------- mpirun detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was: Process name: [[32296,1],1] Exit code: 1 -------------------------------------------------------------------------- (base) [root@cassiopeia storage]#
@zhenghh04 could you please help check this new error, even install is OK, but run still failed. :(
I installed python3.9 and it works. Maybe try to update Python to 3.9 or use conda environment instead of using the default version, which is <3.9
