jsonargparse Overriding issue for optional dataclass argument

🐛 Bug report

When using an optional dataclass, fields specified first will be wiped by later fields specification.

To reproduce

from dataclasses import dataclass

from jsonargparse import ArgumentParser

@dataclass
class A:
    x: int
    y: int

def main():
    parser = ArgumentParser()
    parser.add_argument('--a', type=A | None)
    args = parser.parse_args([
        '--a.x', '1',
        '--a.y', '2',
    ])
    print(args)

if __name__ == '__main__':
    main()

Expected behavior

Should output as if the dataclass parameter is required:

Namespace(a=Namespace(x=1, y=2))

Actual Result

error: Parser key "a":
  Does not validate against any of the Union subtypes
  Subtypes: (<class '__main__.A'>, <class 'NoneType'>)
  Errors:
    - Validation failed: Key "x" is required but not included in config object or its value is None.
    - Expected a <class 'NoneType'>
  Given value type: <class 'jsonargparse._namespace.Namespace'>
  Given value: Namespace(x=None, y=2)

Run with JSONARGPARSE_DEBUG=1:

2023-11-21 18:37:44,314 - ArgumentParser - DEBUG - Loaded parser defaults: Namespace(a=None)
2023-11-21 18:37:44,357 - ArgumentParser - DEBUG - Loaded parser defaults: Namespace(x=None, y=None)
2023-11-21 18:37:44,360 - ArgumentParser - DEBUG - Parsed command line arguments: ['--x=1']
2023-11-21 18:37:44,361 - ArgumentParser - DEBUG - Loaded parser defaults: Namespace(x=None, y=None)
2023-11-21 18:37:44,363 - ArgumentParser - DEBUG - Parsed command line arguments: ['--y=2']
2023-11-21 18:37:44,364 - ArgumentParser - DEBUG - Loaded parser defaults: Namespace(x=None, y=None)
2023-11-21 18:37:44,364 - ArgumentParser - DEBUG - Parsed object: Namespace(x=None, y=2)
2023-11-21 18:37:44,366 - ArgumentParser - ERROR - Validation failed: Key "x" is required but not included in config object or its value is None.
2023-11-21 18:37:44,366 - ArgumentParser - ERROR - Parser key "a":
  Does not validate against any of the Union subtypes
  Subtypes: (<class '__main__.A'>, <class 'NoneType'>)
  Errors:
    - Validation failed: Key "x" is required but not included in config object or its value is None.
    - Expected a <class 'NoneType'>
  Given value type: <class 'jsonargparse._namespace.Namespace'>
  Given value: Namespace(x=None, y=2)
2023-11-21 18:37:44,366 - ArgumentParser - DEBUG - Debug enabled, thus raising exception instead of exit.
Traceback (most recent call last):
  File ".../jsonargparse/_core.py", line 1038, in check_required
    raise TypeError
TypeError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".../jsonargparse/_core.py", line 1079, in check_config
    check_required(cfg, self)
  File ".../jsonargparse/_core.py", line 1040, in check_required
    raise TypeError(
TypeError: Key "x" is required but not included in config object or its value is None.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".../jsonargparse/_core.py", line 441, in parse_object
    parsed_cfg = self._parse_common(
                 ^^^^^^^^^^^^^^^^^^^
  File ".../jsonargparse/_core.py", line 319, in _parse_common
    self.check_config(cfg, skip_required=skip_required)
  File ".../jsonargparse/_core.py", line 1087, in check_config
    raise type(ex)(message) from ex
TypeError: Validation failed: Key "x" is required but not included in config object or its value is None.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".../jsonargparse/_typehints.py", line 695, in adapt_typehints
    vals.append(adapt_typehints(val, subtypehint, **adapt_kwargs))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../jsonargparse/_typehints.py", line 840, in adapt_typehints
    val = parser.parse_object(val, defaults=sub_defaults.get() or list_item)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../jsonargparse/_deprecated.py", line 123, in patched_parse
    cfg = parse_method(*args, _skip_check=_skip_check, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../jsonargparse/_core.py", line 451, in parse_object
    self.error(str(ex), ex)
  File ".../jsonargparse/_core.py", line 1000, in error
    raise argument_error(message) from ex
argparse.ArgumentError: Validation failed: Key "x" is required but not included in config object or its value is None.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".../jsonargparse/_typehints.py", line 498, in _check_type
    raise ex
  File ".../jsonargparse/_typehints.py", line 485, in _check_type
    val = adapt_typehints(val, self._typehint, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../jsonargparse/_typehints.py", line 700, in adapt_typehints
    raise_union_unexpected_value(typehint, val, vals)
  File ".../jsonargparse/_typehints.py", line 590, in raise_union_unexpected_value
    raise ValueError(
ValueError: Does not validate against any of the Union subtypes
Subtypes: (<class '__main__.A'>, <class 'NoneType'>)
Errors:
  - Validation failed: Key "x" is required but not included in config object or its value is None.
  - Expected a <class 'NoneType'>
Given value type: <class 'jsonargparse._namespace.Namespace'>
Given value: Namespace(x=None, y=2)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".../jsonargparse/_core.py", line 1081, in check_config
    check_values(cfg)
  File ".../jsonargparse/_core.py", line 1071, in check_values
    raise ex
  File ".../jsonargparse/_core.py", line 1066, in check_values
    self._check_value_key(action, val, key, ccfg)
  File ".../jsonargparse/_core.py", line 1351, in _check_value_key
    value = action._check_type(value, cfg=cfg)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../jsonargparse/_typehints.py", line 508, in _check_type
    raise TypeError(f'Parser key "{self.dest}"{elem}:\n{error}') from ex
TypeError: Parser key "a":
  Does not validate against any of the Union subtypes
  Subtypes: (<class '__main__.A'>, <class 'NoneType'>)
  Errors:
    - Validation failed: Key "x" is required but not included in config object or its value is None.
    - Expected a <class 'NoneType'>
  Given value type: <class 'jsonargparse._namespace.Namespace'>
  Given value: Namespace(x=None, y=2)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".../jsonargparse/_core.py", line 394, in parse_args
    parsed_cfg = self._parse_common(
                 ^^^^^^^^^^^^^^^^^^^
  File ".../jsonargparse/_core.py", line 319, in _parse_common
    self.check_config(cfg, skip_required=skip_required)
  File ".../jsonargparse/_core.py", line 1087, in check_config
    raise type(ex)(message) from ex
TypeError: Parser key "a":
  Does not validate against any of the Union subtypes
  Subtypes: (<class '__main__.A'>, <class 'NoneType'>)
  Errors:
    - Validation failed: Key "x" is required but not included in config object or its value is None.
    - Expected a <class 'NoneType'>
  Given value type: <class 'jsonargparse._namespace.Namespace'>
  Given value: Namespace(x=None, y=2)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./test.py", line 20, in <module>
    main()
  File "./test.py", line 13, in main
    args = parser.parse_args([
           ^^^^^^^^^^^^^^^^^^^
  File ".../jsonargparse/_deprecated.py", line 123, in patched_parse
    cfg = parse_method(*args, _skip_check=_skip_check, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../jsonargparse/_core.py", line 403, in parse_args
    self.error(str(ex), ex)
  File ".../jsonargparse/_core.py", line 1003, in error
    raise argument_error(message) from ex
argparse.ArgumentError: Parser key "a":
  Does not validate against any of the Union subtypes
  Subtypes: (<class '__main__.A'>, <class 'NoneType'>)
  Errors:
    - Validation failed: Key "x" is required but not included in config object or its value is None.
    - Expected a <class 'NoneType'>
  Given value type: <class 'jsonargparse._namespace.Namespace'>
  Given value: Namespace(x=None, y=2)

Environment

jsonargparse version: 4.27.0
Python version: 3.11.6
How jsonargparse was installed: pip install jsonargparse[signatures, omegaconf]
OS: Linux (Ubuntu 22.04)

Nov 21 '23 10:11 function2-llx

Thank you for reporting! Just to give an update. I did look at this and it is not a trivial fix. I do have a potential idea, but haven't had time for it.

Jan 29 '24 22:01 mauvilsa

are there any updates here?

Mar 29 '24 05:03 eunwoosh

are there any updates here?

No, still the same as my last comment.

Mar 29 '24 10:03 mauvilsa

I checked again the reproduction code and it works as expected now. Seems it was fixed in 4.28.0, though not sure what changed fixed it. Anyway, closing this.

Jun 09 '24 09:06 mauvilsa