pydicom warnings encountered during `test_b0dwi_for_fmap`
When the test_b0dwi_for_fmap test in heudiconv/tests/test_convert.py is run, pydicom emits two UserWarnings with the message "The value length (22) exceeds the maximum length of 16 allowed for VR SH." (presumably because one or more files in heudiconv/tests/data/b0dwiForFmap/ are ill-formed), and then pytest's filterwarnings setting causes these warnings to be converted into errors that are caught & logged here — and yet, the test does not fail.
For the record, the output from running just this test as of commit 2c6d228 is:
============================= test session starts ==============================
platform darwin -- Python 3.11.3, pytest-7.3.0, pluggy-1.0.0 -- /Users/jwodder/dartmouth/heudiconv/.tox/py3/bin/python
cachedir: .tox/py3/.pytest_cache
rootdir: /Users/jwodder/dartmouth/heudiconv
configfile: tox.ini
collecting ... collected 120 items / 119 deselected / 1 selected
heudiconv/tests/test_convert.py::test_b0dwi_for_fmap WARNING: Heuristic is missing an `infotoids` method, assigning empty method and using provided subject id b0dwiForFmap. Provide `session` and `locator` fields for best results.
230412-10:38:52,754 nipype.workflow INFO:
[Node] Setting-up "convert" in "/private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/dcm2niix25vaxluk/convert".
INFO: [Node] Setting-up "convert" in "/private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/dcm2niix25vaxluk/convert".
230412-10:38:52,762 nipype.workflow INFO:
[Node] Executing "convert" <nipype.interfaces.dcm2nii.Dcm2niix>
INFO: [Node] Executing "convert" <nipype.interfaces.dcm2nii.Dcm2niix>
230412-10:38:52,949 nipype.interface INFO:
stdout 2023-04-12T10:38:52.949521:Compression will be faster with 'pigz' installed http://macappstore.org/pigz/
INFO: stdout 2023-04-12T10:38:52.949521:Compression will be faster with 'pigz' installed http://macappstore.org/pigz/
230412-10:38:52,950 nipype.interface INFO:
stdout 2023-04-12T10:38:52.949521:Chris Rorden's dcm2niiX version v1.0.20220720 Clang13.0.0 x86-64 (64-bit MacOS)
230412-10:38:52,950 nipype.interface INFO:
stdout 2023-04-12T10:38:52.949521:Found 3 DICOM file(s)
230412-10:38:52,950 nipype.interface INFO:
stdout 2023-04-12T10:38:52.949521:Convert 3 DICOM as /private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/pytest-of-jwodder/pytest-256/test_b0dwi_for_fmap0/sub-b0dwiForFmap/fmap/sub-b0dwiForFmap_acq-b0dwi_epi_heudiconv021 (150x152x81x3)
INFO: stdout 2023-04-12T10:38:52.949521:Chris Rorden's dcm2niiX version v1.0.20220720 Clang13.0.0 x86-64 (64-bit MacOS)
INFO: stdout 2023-04-12T10:38:52.949521:Found 3 DICOM file(s)
INFO: stdout 2023-04-12T10:38:52.949521:Convert 3 DICOM as /private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/pytest-of-jwodder/pytest-256/test_b0dwi_for_fmap0/sub-b0dwiForFmap/fmap/sub-b0dwiForFmap_acq-b0dwi_epi_heudiconv021 (150x152x81x3)
230412-10:38:53,606 nipype.interface INFO:
stdout 2023-04-12T10:38:53.606512:Conversion required 0.716647 seconds (0.715186 for core code).
INFO: stdout 2023-04-12T10:38:53.606512:Conversion required 0.716647 seconds (0.715186 for core code).
230412-10:38:53,642 nipype.workflow INFO:
[Node] Finished "convert", elapsed time 0.78719s.
INFO: [Node] Finished "convert", elapsed time 0.78719s.
WARNING: Diffusion-weighted image saved in non dwi folder (/private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/pytest-of-jwodder/pytest-256/test_b0dwi_for_fmap0/sub-b0dwiForFmap/fmap)
WARNING: .bvec and .bval files will be generated. This is NOT BIDS compliant
230412-10:38:53,660 nipype.workflow INFO:
[Node] Setting-up "embedder" in "/private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/embedmeta3r_3optu/embedder".
INFO: [Node] Setting-up "embedder" in "/private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/embedmeta3r_3optu/embedder".
230412-10:38:53,665 nipype.workflow INFO:
[Node] Executing "embedder" <nipype.interfaces.utility.wrappers.Function>
INFO: [Node] Executing "embedder" <nipype.interfaces.utility.wrappers.Function>
230412-10:38:53,689 nipype.workflow INFO:
[Node] Finished "embedder", elapsed time 0.021059s.
230412-10:38:53,689 nipype.workflow WARNING:
Storing result file without outputs
INFO: [Node] Finished "embedder", elapsed time 0.021059s.
WARNING: Storing result file without outputs
230412-10:38:53,690 nipype.workflow WARNING:
[Node] Error on "embedder" (/private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/embedmeta3r_3optu/embedder)
WARNING: [Node] Error on "embedder" (/private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/embedmeta3r_3optu/embedder)
ERROR: Embedding failed: Exception raised while executing Node embedder.
Traceback:
Traceback (most recent call last):
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/tag.py", line 28, in tag_in_exception
yield
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataset.py", line 2389, in walk
data_element = self[tag]
~~~~^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataset.py", line 939, in __getitem__
self[tag] = DataElement_from_raw(elem, character_set, self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataelem.py", line 859, in DataElement_from_raw
value = convert_value(vr, raw, encoding)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 768, in convert_value
return converter(byte_string, encodings, VR)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 527, in convert_text
as_strings = [convert_single_string(value, encodings, vr)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 527, in <listcomp>
as_strings = [convert_single_string(value, encodings, vr)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 557, in convert_single_string
validate_value(
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/valuerep.py", line 290, in validate_value
warnings.warn(msg)
UserWarning: The value length (22) exceeds the maximum length of 16 allowed for VR SH.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run
runtime = self._run_interface(runtime)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/nipype/interfaces/utility/wrappers.py", line 142, in _run_interface
out = function_handle(**args)
^^^^^^^^^^^^^^^^^^^^^^^
File "<string>", line 30, in embed_dicom_and_nifti_metadata
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/dcmstack/dcmstack.py", line 1156, in parse_and_stack
results = parse_and_group(src_paths,
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/dcmstack/dcmstack.py", line 1060, in parse_and_group
meta = extractor(dcm)
^^^^^^^^^^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/dcmstack/extract.py", line 455, in __call__
dcm.decode()
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataset.py", line 551, in decode
self.walk(decode_callback, recursive=False)
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataset.py", line 2388, in walk
with tag_in_exception(tag):
File "/usr/local/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/contextlib.py", line 155, in __exit__
self.gen.throw(typ, value, traceback)
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/tag.py", line 32, in tag_in_exception
raise type(exc)(msg) from exc
UserWarning: With tag (0051, 100e) got exception: The value length (22) exceeds the maximum length of 16 allowed for VR SH.
Traceback (most recent call last):
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/tag.py", line 28, in tag_in_exception
yield
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataset.py", line 2389, in walk
data_element = self[tag]
~~~~^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataset.py", line 939, in __getitem__
self[tag] = DataElement_from_raw(elem, character_set, self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataelem.py", line 859, in DataElement_from_raw
value = convert_value(vr, raw, encoding)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 768, in convert_value
return converter(byte_string, encodings, VR)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 527, in convert_text
as_strings = [convert_single_string(value, encodings, vr)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 527, in <listcomp>
as_strings = [convert_single_string(value, encodings, vr)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 557, in convert_single_string
validate_value(
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/valuerep.py", line 290, in validate_value
warnings.warn(msg)
UserWarning: The value length (22) exceeds the maximum length of 16 allowed for VR SH.
230412-10:38:53,694 nipype.workflow INFO:
[Node] Setting-up "convert" in "/private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/dcm2niixo_3eh826/convert".
INFO: [Node] Setting-up "convert" in "/private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/dcm2niixo_3eh826/convert".
230412-10:38:53,698 nipype.workflow INFO:
[Node] Executing "convert" <nipype.interfaces.dcm2nii.Dcm2niix>
INFO: [Node] Executing "convert" <nipype.interfaces.dcm2nii.Dcm2niix>
230412-10:38:53,794 nipype.interface INFO:
stdout 2023-04-12T10:38:53.793922:Chris Rorden's dcm2niiX version v1.0.20220720 Clang13.0.0 x86-64 (64-bit MacOS)
INFO: stdout 2023-04-12T10:38:53.793922:Chris Rorden's dcm2niiX version v1.0.20220720 Clang13.0.0 x86-64 (64-bit MacOS)
230412-10:38:53,794 nipype.interface INFO:
stdout 2023-04-12T10:38:53.793922:Found 3 DICOM file(s)
230412-10:38:53,794 nipype.interface INFO:
stdout 2023-04-12T10:38:53.793922:Convert 3 DICOM as /private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/pytest-of-jwodder/pytest-256/test_b0dwi_for_fmap0/sub-b0dwiForFmap/dwi/sub-b0dwiForFmap_acq-b0dwi_dwi_heudiconv106 (150x152x81x3)
INFO: stdout 2023-04-12T10:38:53.793922:Found 3 DICOM file(s)
INFO: stdout 2023-04-12T10:38:53.793922:Convert 3 DICOM as /private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/pytest-of-jwodder/pytest-256/test_b0dwi_for_fmap0/sub-b0dwiForFmap/dwi/sub-b0dwiForFmap_acq-b0dwi_dwi_heudiconv106 (150x152x81x3)
230412-10:38:54,437 nipype.interface INFO:
stdout 2023-04-12T10:38:54.437402:Conversion required 0.701784 seconds (0.697725 for core code).
INFO: stdout 2023-04-12T10:38:54.437402:Conversion required 0.701784 seconds (0.697725 for core code).
230412-10:38:54,473 nipype.workflow INFO:
[Node] Finished "convert", elapsed time 0.773722s.
INFO: [Node] Finished "convert", elapsed time 0.773722s.
230412-10:38:54,487 nipype.workflow INFO:
[Node] Setting-up "embedder" in "/private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/embedmetaw3s_wq60/embedder".
INFO: [Node] Setting-up "embedder" in "/private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/embedmetaw3s_wq60/embedder".
230412-10:38:54,492 nipype.workflow INFO:
[Node] Executing "embedder" <nipype.interfaces.utility.wrappers.Function>
INFO: [Node] Executing "embedder" <nipype.interfaces.utility.wrappers.Function>
230412-10:38:54,509 nipype.workflow INFO:
[Node] Finished "embedder", elapsed time 0.016025s.
INFO: [Node] Finished "embedder", elapsed time 0.016025s.
WARNING: Storing result file without outputs
230412-10:38:54,509 nipype.workflow WARNING:
Storing result file without outputs
230412-10:38:54,511 nipype.workflow WARNING:
[Node] Error on "embedder" (/private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/embedmetaw3s_wq60/embedder)
WARNING: [Node] Error on "embedder" (/private/var/folders/l7/wrkq93d133d8zpn36fmqrq0r0000gn/T/embedmetaw3s_wq60/embedder)
ERROR: Embedding failed: Exception raised while executing Node embedder.
Traceback:
Traceback (most recent call last):
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/tag.py", line 28, in tag_in_exception
yield
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataset.py", line 2389, in walk
data_element = self[tag]
~~~~^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataset.py", line 939, in __getitem__
self[tag] = DataElement_from_raw(elem, character_set, self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataelem.py", line 859, in DataElement_from_raw
value = convert_value(vr, raw, encoding)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 768, in convert_value
return converter(byte_string, encodings, VR)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 527, in convert_text
as_strings = [convert_single_string(value, encodings, vr)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 527, in <listcomp>
as_strings = [convert_single_string(value, encodings, vr)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 557, in convert_single_string
validate_value(
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/valuerep.py", line 290, in validate_value
warnings.warn(msg)
UserWarning: The value length (22) exceeds the maximum length of 16 allowed for VR SH.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run
runtime = self._run_interface(runtime)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/nipype/interfaces/utility/wrappers.py", line 142, in _run_interface
out = function_handle(**args)
^^^^^^^^^^^^^^^^^^^^^^^
File "<string>", line 30, in embed_dicom_and_nifti_metadata
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/dcmstack/dcmstack.py", line 1156, in parse_and_stack
results = parse_and_group(src_paths,
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/dcmstack/dcmstack.py", line 1060, in parse_and_group
meta = extractor(dcm)
^^^^^^^^^^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/dcmstack/extract.py", line 455, in __call__
dcm.decode()
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataset.py", line 551, in decode
self.walk(decode_callback, recursive=False)
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataset.py", line 2388, in walk
with tag_in_exception(tag):
File "/usr/local/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/contextlib.py", line 155, in __exit__
self.gen.throw(typ, value, traceback)
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/tag.py", line 32, in tag_in_exception
raise type(exc)(msg) from exc
UserWarning: With tag (0051, 100e) got exception: The value length (22) exceeds the maximum length of 16 allowed for VR SH.
Traceback (most recent call last):
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/tag.py", line 28, in tag_in_exception
yield
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataset.py", line 2389, in walk
data_element = self[tag]
~~~~^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataset.py", line 939, in __getitem__
self[tag] = DataElement_from_raw(elem, character_set, self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/dataelem.py", line 859, in DataElement_from_raw
value = convert_value(vr, raw, encoding)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 768, in convert_value
return converter(byte_string, encodings, VR)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 527, in convert_text
as_strings = [convert_single_string(value, encodings, vr)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 527, in <listcomp>
as_strings = [convert_single_string(value, encodings, vr)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/values.py", line 557, in convert_single_string
validate_value(
File "/Users/jwodder/dartmouth/heudiconv/.tox/py3/lib/python3.11/site-packages/pydicom/valuerep.py", line 290, in validate_value
warnings.warn(msg)
UserWarning: The value length (22) exceeds the maximum length of 16 allowed for VR SH.
PASSED
====================== 1 passed, 119 deselected in 3.47s =======================
py3: OK (13.04=setup[6.54]+cmd[6.50] seconds)
congratulations :) (13.10 seconds)
just to make sure -- we have two issues
- likely a "bad" test file which causes the warning. Fix: replace the file or fix the pydicom if warning is "wrong", or just indeed keep the "bad" file as a test for this specific warning to be just a warning and not an error causing the whole conversion to fail. (edit: after all - documented bug is a feature! ;))
- pytest does not error out although should have. Fix: troubleshoot why + fix (likely in pytest)
correct?
@yarikoptic The first item is correct. As for the second, pytest does convert the warning into an exception, but that exception is then caught by this code. I think the more pertinent issue is that the test does not fail.
Ah, thank you for digesting it for me! We indeed just allow for embedder to fail and issue a warning in normal operations. We might disable that during testing to thus allow such exceptions to bubble up.