CLEAN: Reconcile types in pandas._typing.py and pandas-stubs._typing.pyi

[Dr-Irv]

There are some inconsistencies in types in pandas._typing.py and pandas-stubs._typing.pyi . These should be reconciled.

In addition, types that are only in the stubs should probably be private with a leading underscore _

[bashtage]

Are there any tools that will help keep stubs in sync with pandas? When I went through io/stata.py there were a lot of changes since these were originally collatted.

[Dr-Irv]

Are there any tools that will help keep stubs in sync with pandas? When I went through io/stata.py there were a lot of changes since these were originally collatted.

Not that I'm aware of. Maybe stubtest could help

[twoertwein]

Might not make sense to fix all of them (some exist on main but not in 1.4.x) but it gives a quick overview what could be changed poe stubtest | grep "pandas\._typing"

pandas._typing.AggFuncTypeDict is not a type pandas._typing.Axes is not a Union pandas._typing.AxisType is not present at runtime pandas._typing.BaseBuffer.init is inconsistent, stub does not have *args argument "args" pandas._typing.Buffer is not present at runtime pandas._typing.CompressionDict is not a type pandas._typing.ConvertersArg is not present at runtime pandas._typing.DateTimeErrorChoices is not present at runtime pandas._typing.DatetimeLikeScalar is not a TypeVar pandas._typing.DtypeNp is not present at runtime pandas._typing.FileOrBuffer is not present at runtime pandas._typing.FilePathOrBuffer is not present at runtime pandas._typing.FilePathOrBytesBuffer is not present at runtime pandas._typing.FrameOrSeries is not present at runtime pandas._typing.FrameOrSeriesUnion is not present at runtime pandas._typing.FuncType is not a recognised type alias pandas._typing.Function is not present at runtime pandas._typing.GroupByObject is not present at runtime pandas._typing.GroupByObjectNonScalar is not present at runtime pandas._typing.HashableT is not present at runtime pandas._typing.IgnoreRaise is not present at runtime pandas._typing.IndexType is not present at runtime pandas._typing.IndexingInt is not present at runtime pandas._typing.IntervalClosedType is not present at runtime pandas._typing.KeysArgType is not present at runtime pandas._typing.Label is not present at runtime pandas._typing.ListLike is not present at runtime pandas._typing.MaskType is not present at runtime pandas._typing.ParseDatesArg is not present at runtime pandas._typing.ReadBuffer.init is inconsistent, stub does not have *args argument "args" pandas._typing.S1 is not present at runtime pandas._typing.ScalarT is not present at runtime pandas._typing.SeriesAxisType is not present at runtime pandas._typing.StrLike is not present at runtime pandas._typing.T1 is not present at runtime pandas._typing.T2 is not present at runtime pandas._typing.WriteBuffer.init is inconsistent, stub does not have *args argument "args" pandas._typing.XMLParsers is not a Union pandas._typing.all is not present at runtime pandas._typing.np_ndarray_anyint is not present at runtime pandas._typing.np_ndarray_bool is not present at runtime pandas._typing.np_ndarray_int is not present at runtime pandas._typing.np_ndarray_int16 is not present at runtime pandas._typing.np_ndarray_int32 is not present at runtime pandas._typing.np_ndarray_int64 is not present at runtime pandas._typing.np_ndarray_int8 is not present at runtime pandas._typing.np_ndarray_str is not present at runtime pandas._typing.np_ndarray_uint is not present at runtime pandas._typing.np_ndarray_uint16 is not present at runtime pandas._typing.np_ndarray_uint32 is not present at runtime pandas._typing.np_ndarray_uint64 is not present at runtime pandas._typing.np_ndarray_uint8 is not present at runtime pandas._typing.num is not present at runtime pandas._typing.type_t is not a type

[Dr-Irv]

Can this tell us which ones are in pandas._typing.py, but not pandas._typing.pyi ?

Also would be best if we could use what is in main from pandas just to keep things moving forward.

[twoertwein]

Can this tell us which ones are in pandas._typing.py, but not pandas._typing.pyi ?

I think so, would probably need to remove --ignore-missing-stub from the stubtest call.

Also would be best if we could use what is in main from pandas just to keep things moving forward.

stubtest will use whichever version of pandas that is installed. Currently, that is 1.4.x. I think pandas has nightly builds, we could install that in pyproject.toml instead of 1.4.x: but then we might get some "troubles" with pytests, more deprecation warnings (before 1.5) and breaking changes (after 1.5 is released).

[twoertwein]

Can this tell us which ones are in pandas._typing.py, but not pandas._typing.pyi ?

I wrote some quick&ugly ast parsing code to check when pandas-stubs has annotations but pandas has none (I have been using it for https://github.com/pandas-dev/pandas/pull/47827 and similar PRs). Could extend that, but stubtest without --ignore-missing-stub would be a better choice.

[Dr-Irv]

stubtest will use whichever version of pandas that is installed. Currently, that is 1.4.x. I think pandas has nightly builds, we could install that in pyproject.toml instead of 1.4.x: but then we might get some "troubles" with pytests, more deprecation warnings (before 1.5) and breaking changes (after 1.5 is released).

Maybe you could just figure out a way to run stubtest (but not mypy, pytest, etc.) against the current version in pandas/main so we can start reconciling the types. There is good typing work in pandas that we can pull over to the stubs without making the stubs incorrect for 1.4.x

[twoertwein]

Could have a step ala "remove pandas 1.4x. and install pandas nightly" with an appropriate rollback.