mypy icon indicating copy to clipboard operation
mypy copied to clipboard
verified publisher icon python

WIP: adds better `namedtuple` semantic analyzer

Open sobolevn opened this issue 4 years ago • 26 comments

This is almost a complete rewrite of inline NamedTuple and namedtuple semantic analyzer.

Changelog:

  1. NamedTuple now supports **kwargs as it should
  2. NamedTuple and namedtuple now have better validation for fields: keywords, underscores, duplication: basically the same as collections.namedtuple does at runtime
  3. namedtuple now supports rename argument (it actually renames invalid fields)
  4. We now also check NamedTuple() and namedtuple() calls during typechecking, this allows us to simplify the setup
  5. Namedtuples no longer think that bytes are valid arguments on python3
  6. NamedTuple and namedtuple now support named arguments as they should
  7. I've moved all errors to a single place, now semanal.py does not raise any namedtuple-expr-related errors
  8. More consistent errors: I've copied texts from runtime version of collections.namedtuple, added "" everywhere

This is still WIP, but I've wrote several tests for this in testsemanal.py. They are passing. And self check also passes.

Снимок экрана 2021-09-27 в 18 02 24

I can continue to work on this after #11162 is merged or rejected. Because I need better typing fixtures for NamedTuple type.

Closes #11047

sobolevn avatar Sep 27 '21 15:09 sobolevn

Diff from mypy_primer, showing the effect of this PR on open source code:

optuna (https://github.com/optuna/optuna.git)
- tests/integration_tests/test_chainer.py:83: error: First argument to namedtuple() should be "MockTrainer", not "_MockTrainer"
+ tests/integration_tests/test_chainer.py:83: error: First argument to "namedtuple()" should be "MockTrainer", not "_MockTrainer"
- tests/integration_tests/test_chainer.py:84: error: First argument to namedtuple() should be "MockUpdater", not "_MockUpdater"
+ tests/integration_tests/test_chainer.py:84: error: First argument to "namedtuple()" should be "MockUpdater", not "_MockUpdater"
- tests/integration_tests/test_chainer.py:97: error: First argument to namedtuple() should be "MockTrainer", not "_MockTrainer"
+ tests/integration_tests/test_chainer.py:97: error: First argument to "namedtuple()" should be "MockTrainer", not "_MockTrainer"

pandas (https://github.com/pandas-dev/pandas.git)
+ pandas/core/frame.py:1355: error: unused "type: ignore" comment
+ pandas/core/frame.py:1356: error: "namedtuple()" expects a string literal as the typename argument  [misc]

manticore (https://github.com/trailofbits/manticore.git)
- manticore/native/cpu/x86.py:147: error: First argument to namedtuple() should be "Regspec", not "RegSpec"
+ manticore/native/cpu/x86.py:147: error: First argument to "namedtuple()" should be "Regspec", not "RegSpec"
- manticore/native/cpu/aarch64.py:44: error: First argument to namedtuple() should be "Condspec", not "CondSpec"
+ manticore/native/cpu/aarch64.py:44: error: First argument to "namedtuple()" should be "Condspec", not "CondSpec"
- manticore/native/cpu/aarch64.py:79: error: First argument to namedtuple() should be "Regspec", not "RegSpec"
+ manticore/native/cpu/aarch64.py:79: error: First argument to "namedtuple()" should be "Regspec", not "RegSpec"

paasta (https://github.com/yelp/paasta.git)
- paasta_tools/iptables.py:23: error: First argument to namedtuple() should be "_RuleBase", not "Rule"
+ paasta_tools/iptables.py:23: error: First argument to "namedtuple()" should be "_RuleBase", not "Rule"

github-actions[bot] avatar Sep 27 '21 15:09 github-actions[bot]

Diff from mypy_primer, showing the effect of this PR on open source code:

optuna (https://github.com/optuna/optuna.git)
- tests/integration_tests/test_chainer.py:83: error: First argument to namedtuple() should be "MockTrainer", not "_MockTrainer"
+ tests/integration_tests/test_chainer.py:83: error: First argument to "namedtuple()" should be "MockTrainer", not "_MockTrainer"
- tests/integration_tests/test_chainer.py:84: error: First argument to namedtuple() should be "MockUpdater", not "_MockUpdater"
+ tests/integration_tests/test_chainer.py:84: error: First argument to "namedtuple()" should be "MockUpdater", not "_MockUpdater"
- tests/integration_tests/test_chainer.py:97: error: First argument to namedtuple() should be "MockTrainer", not "_MockTrainer"
+ tests/integration_tests/test_chainer.py:97: error: First argument to "namedtuple()" should be "MockTrainer", not "_MockTrainer"

manticore (https://github.com/trailofbits/manticore.git)
- manticore/native/cpu/x86.py:147: error: First argument to namedtuple() should be "Regspec", not "RegSpec"
+ manticore/native/cpu/x86.py:147: error: First argument to "namedtuple()" should be "Regspec", not "RegSpec"
- manticore/native/cpu/aarch64.py:44: error: First argument to namedtuple() should be "Condspec", not "CondSpec"
+ manticore/native/cpu/aarch64.py:44: error: First argument to "namedtuple()" should be "Condspec", not "CondSpec"
- manticore/native/cpu/aarch64.py:79: error: First argument to namedtuple() should be "Regspec", not "RegSpec"
+ manticore/native/cpu/aarch64.py:79: error: First argument to "namedtuple()" should be "Regspec", not "RegSpec"

pandas (https://github.com/pandas-dev/pandas.git)
+ pandas/core/frame.py:1355: error: unused "type: ignore" comment
+ pandas/core/frame.py:1356: error: "namedtuple()" expects a string literal as the typename argument  [misc]

paasta (https://github.com/yelp/paasta.git)
- paasta_tools/iptables.py:23: error: First argument to namedtuple() should be "_RuleBase", not "Rule"
+ paasta_tools/iptables.py:23: error: First argument to "namedtuple()" should be "_RuleBase", not "Rule"

github-actions[bot] avatar Sep 27 '21 15:09 github-actions[bot]

This PR would also help us a lot: https://github.com/python/typeshed/pull/6080

Since we now typecheck NamedTupleExpr.call, together with that PR it can detect cases, when we use invalid kwargs and fields combination.

sobolevn avatar Sep 27 '21 15:09 sobolevn

Diff from mypy_primer, showing the effect of this PR on open source code:

optuna (https://github.com/optuna/optuna.git)
- tests/integration_tests/test_chainer.py:83: error: First argument to namedtuple() should be "MockTrainer", not "_MockTrainer"
+ tests/integration_tests/test_chainer.py:83: error: First argument to "namedtuple()" should be "MockTrainer", not "_MockTrainer"
- tests/integration_tests/test_chainer.py:84: error: First argument to namedtuple() should be "MockUpdater", not "_MockUpdater"
+ tests/integration_tests/test_chainer.py:84: error: First argument to "namedtuple()" should be "MockUpdater", not "_MockUpdater"
- tests/integration_tests/test_chainer.py:97: error: First argument to namedtuple() should be "MockTrainer", not "_MockTrainer"
+ tests/integration_tests/test_chainer.py:97: error: First argument to "namedtuple()" should be "MockTrainer", not "_MockTrainer"

pandas (https://github.com/pandas-dev/pandas.git)
+ pandas/core/frame.py:1355: error: unused "type: ignore" comment
+ pandas/core/frame.py:1356: error: "namedtuple()" expects a string literal as the typename argument  [misc]

manticore (https://github.com/trailofbits/manticore.git)
- manticore/native/cpu/x86.py:147: error: First argument to namedtuple() should be "Regspec", not "RegSpec"
+ manticore/native/cpu/x86.py:147: error: First argument to "namedtuple()" should be "Regspec", not "RegSpec"
- manticore/native/cpu/aarch64.py:44: error: First argument to namedtuple() should be "Condspec", not "CondSpec"
+ manticore/native/cpu/aarch64.py:44: error: First argument to "namedtuple()" should be "Condspec", not "CondSpec"
- manticore/native/cpu/aarch64.py:79: error: First argument to namedtuple() should be "Regspec", not "RegSpec"
+ manticore/native/cpu/aarch64.py:79: error: First argument to "namedtuple()" should be "Regspec", not "RegSpec"

paasta (https://github.com/yelp/paasta.git)
- paasta_tools/iptables.py:23: error: First argument to namedtuple() should be "_RuleBase", not "Rule"
+ paasta_tools/iptables.py:23: error: First argument to "namedtuple()" should be "_RuleBase", not "Rule"

github-actions[bot] avatar Sep 28 '21 08:09 github-actions[bot]

I finally have the time to continue my work!

All PRs that I needed: https://github.com/python/typeshed/pull/6080 / https://github.com/python/mypy/pull/11303 / https://github.com/python/mypy/pull/11162 are merged!

sobolevn avatar Nov 07 '21 09:11 sobolevn

Diff from mypy_primer, showing the effect of this PR on open source code:

manticore (https://github.com/trailofbits/manticore.git)
- manticore/native/cpu/x86.py:147: error: First argument to namedtuple() should be "Regspec", not "RegSpec"
+ manticore/native/cpu/x86.py:147: error: First argument to "namedtuple()" should be "Regspec", not "RegSpec"
- manticore/native/cpu/aarch64.py:44: error: First argument to namedtuple() should be "Condspec", not "CondSpec"
+ manticore/native/cpu/aarch64.py:44: error: First argument to "namedtuple()" should be "Condspec", not "CondSpec"
- manticore/native/cpu/aarch64.py:79: error: First argument to namedtuple() should be "Regspec", not "RegSpec"
+ manticore/native/cpu/aarch64.py:79: error: First argument to "namedtuple()" should be "Regspec", not "RegSpec"

pandas (https://github.com/pandas-dev/pandas.git)
+ pandas/core/frame.py:1338: error: unused "type: ignore" comment
+ pandas/core/frame.py:1339: error: "namedtuple()" expects a string literal as the typename argument  [misc]

github-actions[bot] avatar Nov 07 '21 09:11 github-actions[bot]

All check-namedtuple.test cases are passing. Still a lot to fix though. Снимок экрана 2021-11-07 в 13 37 57

sobolevn avatar Nov 07 '21 10:11 sobolevn

Diff from mypy_primer, showing the effect of this PR on open source code:

manticore (https://github.com/trailofbits/manticore.git)
- manticore/native/cpu/x86.py:147: error: First argument to namedtuple() should be "Regspec", not "RegSpec"
+ manticore/native/cpu/x86.py:147: error: First argument to "namedtuple()" should be "Regspec", not "RegSpec"
- manticore/native/cpu/aarch64.py:44: error: First argument to namedtuple() should be "Condspec", not "CondSpec"
+ manticore/native/cpu/aarch64.py:44: error: First argument to "namedtuple()" should be "Condspec", not "CondSpec"
- manticore/native/cpu/aarch64.py:79: error: First argument to namedtuple() should be "Regspec", not "RegSpec"
+ manticore/native/cpu/aarch64.py:79: error: First argument to "namedtuple()" should be "Regspec", not "RegSpec"

pandas (https://github.com/pandas-dev/pandas.git)
+ pandas/core/frame.py:1338: error: unused "type: ignore" comment
+ pandas/core/frame.py:1339: error: "namedtuple()" expects a string literal as the typename argument  [misc]

github-actions[bot] avatar Nov 07 '21 10:11 github-actions[bot]

Diff from mypy_primer, showing the effect of this PR on open source code:

pandas (https://github.com/pandas-dev/pandas.git)
+ pandas/core/frame.py:1338: error: unused "type: ignore" comment
+ pandas/core/frame.py:1339: error: "namedtuple()" expects a string literal as the typename argument  [misc]

manticore (https://github.com/trailofbits/manticore.git)
- manticore/native/cpu/x86.py:147: error: First argument to namedtuple() should be "Regspec", not "RegSpec"
+ manticore/native/cpu/x86.py:147: error: First argument to "namedtuple()" should be "Regspec", not "RegSpec"
- manticore/native/cpu/aarch64.py:44: error: First argument to namedtuple() should be "Condspec", not "CondSpec"
+ manticore/native/cpu/aarch64.py:44: error: First argument to "namedtuple()" should be "Condspec", not "CondSpec"
- manticore/native/cpu/aarch64.py:79: error: First argument to namedtuple() should be "Regspec", not "RegSpec"
+ manticore/native/cpu/aarch64.py:79: error: First argument to "namedtuple()" should be "Regspec", not "RegSpec"

github-actions[bot] avatar Nov 07 '21 11:11 github-actions[bot]

Diff from mypy_primer, showing the effect of this PR on open source code:

manticore (https://github.com/trailofbits/manticore)
- manticore/native/cpu/x86.py:147: error: First argument to namedtuple() should be "Regspec", not "RegSpec"
+ manticore/native/cpu/x86.py:147: error: First argument to "namedtuple()" should be "Regspec", not "RegSpec"

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/frame.py:1402: error: Unused "type: ignore" comment
+ pandas/core/frame.py:1403: error: "namedtuple()" expects a string literal as the typename argument  [misc]

github-actions[bot] avatar Dec 03 '21 13:12 github-actions[bot]

Diff from mypy_primer, showing the effect of this PR on open source code:

manticore (https://github.com/trailofbits/manticore)
- manticore/native/cpu/x86.py:147: error: First argument to namedtuple() should be "Regspec", not "RegSpec"
+ manticore/native/cpu/x86.py:147: error: First argument to "namedtuple()" should be "Regspec", not "RegSpec"

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/frame.py:1402: error: Unused "type: ignore" comment
+ pandas/core/frame.py:1403: error: "namedtuple()" expects a string literal as the typename argument  [misc]

github-actions[bot] avatar Dec 03 '21 13:12 github-actions[bot]

Diff from mypy_primer, showing the effect of this PR on open source code:

manticore (https://github.com/trailofbits/manticore)
- manticore/native/cpu/x86.py:147: error: First argument to namedtuple() should be "Regspec", not "RegSpec"
+ manticore/native/cpu/x86.py:147: error: First argument to "namedtuple()" should be "Regspec", not "RegSpec"

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/frame.py:1402: error: Unused "type: ignore" comment
+ pandas/core/frame.py:1403: error: "namedtuple()" expects a string literal as the typename argument  [misc]

github-actions[bot] avatar Dec 03 '21 15:12 github-actions[bot]

Diff from mypy_primer, showing the effect of this PR on open source code:

manticore (https://github.com/trailofbits/manticore)
- manticore/native/cpu/x86.py:147: error: First argument to namedtuple() should be "Regspec", not "RegSpec"
+ manticore/native/cpu/x86.py:147: error: First argument to "namedtuple()" should be "Regspec", not "RegSpec"

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/frame.py:1402: error: Unused "type: ignore" comment
+ pandas/core/frame.py:1403: error: "namedtuple()" expects a string literal as the typename argument  [misc]

github-actions[bot] avatar Dec 03 '21 15:12 github-actions[bot]

Diff from mypy_primer, showing the effect of this PR on open source code:

manticore (https://github.com/trailofbits/manticore)
- manticore/native/cpu/x86.py:147: error: First argument to namedtuple() should be "Regspec", not "RegSpec"
+ manticore/native/cpu/x86.py:147: error: First argument to "namedtuple()" should be "Regspec", not "RegSpec"

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/frame.py:1402: error: Unused "type: ignore" comment
+ pandas/core/frame.py:1403: error: "namedtuple()" expects a string literal as the typename argument  [misc]

github-actions[bot] avatar Dec 04 '21 09:12 github-actions[bot]

Diff from mypy_primer, showing the effect of this PR on open source code:

manticore (https://github.com/trailofbits/manticore)
- manticore/native/cpu/x86.py:147: error: First argument to namedtuple() should be "Regspec", not "RegSpec"
+ manticore/native/cpu/x86.py:147: error: First argument to "namedtuple()" should be "Regspec", not "RegSpec"

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/frame.py:1402: error: Unused "type: ignore" comment
+ pandas/core/frame.py:1403: error: "namedtuple()" expects a string literal as the typename argument  [misc]

github-actions[bot] avatar Dec 04 '21 10:12 github-actions[bot]

Diff from mypy_primer, showing the effect of this PR on open source code:

manticore (https://github.com/trailofbits/manticore)
- manticore/native/cpu/x86.py:147: error: First argument to namedtuple() should be "Regspec", not "RegSpec"
+ manticore/native/cpu/x86.py:147: error: First argument to "namedtuple()" should be "Regspec", not "RegSpec"

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/frame.py:1402: error: Unused "type: ignore" comment
+ pandas/core/frame.py:1403: error: "namedtuple()" expects a string literal as the typename argument  [misc]

github-actions[bot] avatar Dec 04 '21 10:12 github-actions[bot]

Diff from mypy_primer, showing the effect of this PR on open source code:

manticore (https://github.com/trailofbits/manticore)
- manticore/native/cpu/x86.py:147: error: First argument to namedtuple() should be "Regspec", not "RegSpec"
+ manticore/native/cpu/x86.py:147: error: First argument to "namedtuple()" should be "Regspec", not "RegSpec"

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/frame.py:1402: error: Unused "type: ignore" comment
+ pandas/core/frame.py:1403: error: "namedtuple()" expects a string literal as the typename argument  [misc]

github-actions[bot] avatar Dec 04 '21 11:12 github-actions[bot]

Diff from mypy_primer, showing the effect of this PR on open source code:

manticore (https://github.com/trailofbits/manticore)
- manticore/native/cpu/x86.py:147: error: First argument to namedtuple() should be "Regspec", not "RegSpec"
+ manticore/native/cpu/x86.py:147: error: First argument to "namedtuple()" should be "Regspec", not "RegSpec"

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/frame.py:1402: error: Unused "type: ignore" comment
+ pandas/core/frame.py:1403: error: "namedtuple()" expects a string literal as the typename argument  [misc]

github-actions[bot] avatar Dec 04 '21 14:12 github-actions[bot]

Diff from mypy_primer, showing the effect of this PR on open source code:

manticore (https://github.com/trailofbits/manticore)
- manticore/native/cpu/x86.py:147: error: First argument to namedtuple() should be "Regspec", not "RegSpec"
+ manticore/native/cpu/x86.py:147: error: First argument to "namedtuple()" should be "Regspec", not "RegSpec"

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/frame.py:1402: error: Unused "type: ignore" comment
+ pandas/core/frame.py:1403: error: "namedtuple()" expects a string literal as the typename argument  [misc]

github-actions[bot] avatar Dec 04 '21 16:12 github-actions[bot]

Diff from mypy_primer, showing the effect of this PR on open source code:

manticore (https://github.com/trailofbits/manticore)
- manticore/native/cpu/x86.py:147: error: First argument to namedtuple() should be "Regspec", not "RegSpec"
+ manticore/native/cpu/x86.py:147: error: First argument to "namedtuple()" should be "Regspec", not "RegSpec"

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/frame.py:1402: error: Unused "type: ignore" comment
+ pandas/core/frame.py:1403: error: "namedtuple()" expects a string literal as the typename argument  [misc]

github-actions[bot] avatar Dec 04 '21 16:12 github-actions[bot]

You might use WORD_SIZE*1 and WORD_SIZE*2 to fix the failure on win32

97littleleaf11 avatar Dec 07 '21 07:12 97littleleaf11

@97littleleaf11 thank you! Can you please be more specific? 🙂 I don't understand your idea.

sobolevn avatar Dec 07 '21 09:12 sobolevn

@sobolevn You can just replace +4 with + WORD_SIZE*1 and replace +8 with + WORD_SIZE*2 in testTypeAlias_toplevel.

97littleleaf11 avatar Dec 07 '21 09:12 97littleleaf11

Diff from mypy_primer, showing the effect of this PR on open source code:

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/frame.py:1402: error: Unused "type: ignore" comment
+ pandas/core/frame.py:1403: error: "namedtuple()" expects a string literal as the typename argument  [misc]

github-actions[bot] avatar Dec 07 '21 11:12 github-actions[bot]

Diff from mypy_primer, showing the effect of this PR on open source code:

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/frame.py:1402: error: Unused "type: ignore" comment
+ pandas/core/frame.py:1403: error: "namedtuple()" expects a string literal as the typename argument  [misc]

github-actions[bot] avatar Dec 07 '21 13:12 github-actions[bot]

Awesome, thank you, @97littleleaf11 It now works.

I will add more tests for .call type checking. And this will be ready to be reviewed.

sobolevn avatar Dec 07 '21 18:12 sobolevn