python library is missing generated sources for validate.proto causing runtime errors

[marcoferrer]

trafficstars

If a user generates python sources for their protos but does not generate python sources for the validate dependency, the python sources will produce an error at runtime for invalid import

The import statement added by protoc from validate import validate_pb2 as validate_dot_validate__pb2

~It doesn't look like this import is ever actually used.~ Would it make sense to include the generated python sources in the validate wheel that is published?

[marcoferrer]

Update. So it looks like we actually need the import in the file so that the validate library can successfully load the annotations at runtime. In my opinion, this is a sign that the python sources for validate.proto should be included in the python library wheel.

[randori-ddejohn]

It's still not clear to me how to get the validate_pb2 generated properly. I have this invocation in my entrypoint script:

STUBS_DIR_PYTHON=/src/python/path/to/python/project
mkdir -p $STUBS_DIR_PYTHON
python3 -m grpc_tools.protoc \
	-I=/src/api/v1 \
	-I"$GOPATH/pkg/mod/github.com/envoyproxy/[email protected]" \
	-I"/protoc/include/" \
	--python_out=$STUBS_DIR_PYTHON \
	--pyi_out=$STUBS_DIR_PYTHON \
	--grpc_python_out=$STUBS_DIR_PYTHON \
	/src/api/v1/**/*.proto

I am generating code for two separate services, the protobufs for which nested in their respective directories. The generated code correctly aligns with the protobuf directory structure, but the Python files have either this import (in *.pyi files):

from validate import validate_pb2 as _validate_pb2

or this import (in *.py files):

from validate import validate_pb2 as validate_dot_validate__pb2

But this Python package was not generated. I am also generating a Go client with no issues as far as I can tell (e.g., I see my_proto.pb.validate.go).

[marcoferrer]

So i think this issue is specifically for the case where we are not trying to regenerate the sources for validate.proto

I will say that after digging into this a bit more. I dont think we want to include the validate_pb2 sources in the “validate” python lib. But instead publish the generated source as a separate lib. This would make it less likely to impact existing users while allowing intentional opt in of using the published generated source. It also protects from users that rely on an older version of the protobuf lib from breaking if we were to publish generated sources using a newer version of the compiler

[gnuletik]

Publishing validate.proto's python sources would be really helpful! I agree that publishing it in a separate package would be best to avoid impacting existing users. Is there anything holding it?

[WAYLON]

I also have the same problem

[ehwjh2010]

I also have the same problem， Is there a solution？

[gnuletik]

@ehwjh2010 It seems that the current workaround for this issue is to manually add a validate python package in your generated grpc package.

Here's the Taskfile section we run after calling buf generate.

  install-python:
    cmds:
      - pip3 install grpcio-tools
      - pip3 install googleapis-common-protos

  # validate.proto is not included inside protoc-gen-validate's python lib
  # https://github.com/bufbuild/protoc-gen-validate/issues/779
  # we need to compile it manually, for Python only
  python-validate:
    vars:
      SRC: https://raw.githubusercontent.com/bufbuild/protoc-gen-validate/main/validate/validate.proto
      DEPS: "./deps"
      DEST: gen/python
    cmds:
      - rm -rf {{ .DEPS }}
      - rm -rf {{ .DEST }}
      - mkdir -p {{ .DEPS }}/validate
      - curl {{ .SRC}} -o {{ .DEPS }}/validate/validate.proto
      - mkdir -p {{ .DEST }}
      - python3 -m grpc_tools.protoc --proto_path={{ .DEPS }} --python_out={{ .DEST }} --grpc_python_out={{ .DEST }} {{ .DEPS }}/validate/validate.proto
      - rm -rf {{ .DEPS }}

[nitlev]

I will say that after digging into this a bit more. I dont think we want to include the validate_pb2 sources in the “validate” python lib. But instead publish the generated source as a separate lib.

And ideally you could pull the source with the library as an extra (ie: pip install protoc-gen-validate[sources]) !

[ayberkozgur]

Any update on this?

[chrispine]

Hello, thanks for your interest! As explained in the README, this project is in maintenance mode. We recommend upgrading to protovalidate.

I'm going to go ahead and close this issue, but feel free to open one against protovalidate if needed. Thanks!