types-protobuf fails pre-commit cache

[Itay4]

when configuring pre-commit mypy to run with types-protobuf, for example:

  -   repo: https://github.com/pre-commit/mirrors-mypy
      rev: v1.8.0
      hooks:
        -   id: mypy
            additional_dependencies:
              - types-protobuf

it fails with the following:

mypy.....................................................................Failed
- hook id: mypy
- exit code: 2

mypy: can't read file '/home/runner/.cache/pre-commit/repoem44ysms/py_env-python3.10/lib/python3.10/site-packages//google': No such file or directory

i suspect its related to https://github.com/python/typeshed/pull/11249, but not sure - @AlexWaygood @hauntsaninja i would appreciate your insights here

[andylamp]

Encountering the same issue as well, this was previously working....

edit: happy to provide logs etc if that helps, but the resulting message I get when I run the pre-commits is the same as @Itay4 's output shown above.

edit2: if it helps, I tried using both Python 3.10 and Python 3.11 virtual envs for running the pre-commit hooks but the same error occurs.

edit3: I cannot find a way to use a "downgraded" of an additional_dependencies entry in mypy as a pre-commit hook so cannot check that previous versions worked, but we certainly did not have this problem previously...

[srittau]

Neither #11249, not #11245, the two PRs included in the latest release, should have an influence here. I'm not sure what exactly is going on here, but types-protobuf doesn't add a google directory, but a google-stubs directory, so it's no wonder that mypy can't find it. Does downgrading to types-protobuf 4.24.0.4 help?

[andylamp]

Hi @srittau, as I mentioned previously I do know a way to specific specific version within mypy in a pre-commit environment.

Using the default though, exhibits the behavior mentioned by the original author.

If you know a way to "pin" an additional depency version, let me know and I can test it.

[srittau]

This problem was introduced by typeshed-internal/stub_uploader#109. Previously, the google-stubs directory was missing the py.typed file that marked the stubs as partial. I think this was an error according to the typing spec:

For namespace packages (see PEP 420), stub-only packages should use the -stubs suffix on only the root namespace package. All stub-only namespace packages should omit __init__.pyi files. py.typed marker files are not necessary for stub-only packages, but similarly to packages with inline types, if used, they should be in submodules of the namespace to avoid conflicts and for clarity.

Cc @Avasam

[Avasam]

If I understand correctly, "google" is a namespace package and the py.typed marker should be found in google-stubs/protobuf/ rather than google-stubs/ ? (that's what I'm getting from the part of the spec srittau is quotig)

In the short term, simply setting partial_stub to false in https://github.com/python/typeshed/blob/main/stubs/protobuf/METADATA.toml (and namespace packages in general) with an appropriate comment should be enough to fix the current issue (no py.typed file should be generated for non-partial stubs, as it is optional and redundant as per the spec).

In the longer term, we may want a way to identify those so stub_uploader adds the py.typed file to the submodules. Like the lack of any file at the top-level.

[andylamp]

@Avasam thanks for the explanation, any workarounds for this issue for the interim :-)?

[srittau]

If I understand correctly, "google" is a namespace package and the py.typed marker should be found in google-stubs/protobuf/ rather than google-stubs/ ? (that's what I'm getting from the part of the spec srittau is quotig)

That's my understanding as well. I think making the necessary changes to the stub_uploader should be fairly simple, although I haven't investigated deeply.

[Avasam]

@Avasam thanks for the explanation, any workarounds for this issue for the interim :-)?

@andylamp Check the very bottom of https://github.com/pre-commit/mirrors-mypy 's readme. I'm pretty sure it's just using pip, so the same markers should work to pin to an older version: types-protobuf==X.X.X

[tonkolviktor]

Hi, would it be possible to delete this version from pypi?

[srittau]

I'm not sure who has access to the packages on PyPI. I think @ilevkivskyi could do it.

[alanbchristie]

Also finding all my CI pipelines have broken. In my GitLab CI script I have had to pin it to 4.24.0.4 with the following: -

- repo: https://github.com/pre-commit/mirrors-mypy
  rev: v1.8.0
  hooks:
  - id: mypy
    files: ^app
    args:
    - --ignore-missing-imports
    additional_dependencies:
    - types-protobuf==4.24.0.4
    - types-PyYAML
    - types-python-dateutil
    - types-pytz
    - types-requests

Any chance this release can be pulled very soon?

I would suggest that publishing what looks like a nightly build to PyPI (4.24.0.20240106) is probably not recommended in the future?

[Avasam]

I would suggest that publishing what looks like a nightly build to PyPI (4.24.0.20240106) is probably not recommended in the future?

Stubs are now calendar-versionned, (hence the bump from 4, an index, to 20240106, a date) with the express purpose to communicate that the version number is not semantic (and has never been). There's no such thing as a major/minor/patch/nightly update in typeshed's stubs. See https://github.com/python/typeshed/issues/10837, https://github.com/python/typeshed/issues/6843 and https://github.com/typeshed-internal/stub_uploader/pull/105 for more details.

I have had to pin it to 4.24.0.4 with the following: - Any chance this release can be pulled very soon?

Hopefully? But I'd recommend pinning your libraries versions either way if you want to avoid unexpected CI breakage. Even an update that fixes or improves existing stubs, or an update to type-checkers, could cause your CI to fail all of a sudden (like new false-positives, using new features your type-checker version doesn't support, fixing a false-positive you had suppressed, etc.)

[ilevkivskyi]

Do you still need me to delete some package from PyPI? If yes, could you please post exactly what is needed (i.e. library and version)?

[alanbchristie]

Hi @ilevkivskyi - For me? I fix the problems with types-protobuf==4.24.0.4 as 4.24.0.20240106 breaks mypy (see #11263).

That's (obviously) not a solution to the problem.

I have a work-around and I don't have a strong feeling on the topic (it's not my repo) so I'm fine - but it means the latest release can't be used by me (and, I suspect, others) because of the problems it seems to cause with mypy.

If someone thinks the problem with types-protobuf is caused by types-protobuf then I guess, yes, pull it. Does someone know what the problem is?

[tonkolviktor]

Hi @ilevkivskyi , the removal idea come from me, IMHO if we know that the release is broken and the fix is not coming in hours, then it should be removed: types-protobuf 4.24.0.20240106 buuut I'm just a guy from the internet :)

[Avasam]

Update to those following this issue: with https://github.com/typeshed-internal/stub_uploader/pull/122, the latest version of types-protobuff (4.24.0.20240129) shouldn't have this issue as it doesn't have any py.typed marker.

The marker file should properly be re-introduced in the subpackages (rather than the root of the namespace package), in a future update to stub_uploader.

[Avasam]

This issue should be resolved and we haven't had any new complaints since then. For the missing py.typed marker in namespace packages, please refer to https://github.com/typeshed-internal/stub_uploader/issues/121#issuecomment-1936357077 for tracking.