Document how to properly package generated python code

[maxnoe]

What language does this apply to?

Python

Describe the problem you are trying to solve.

Shipping a python module that includes generated python code from proto definitions.

The proto definitions are not under the direct control of the authors of the python package.

Describe the solution you'd like

Documentation, and maybe a real fix for #1491 .

Describe alternatives you've considered

Additional context

#1491 contains everything but a real solution to the issue.

The documentation for how to use protobuf with python does not mention packaging or shipping python packages that include generated protobuf code.

There are no guidelines how this could be done and naïve approaches lead to the many different proto structure and import related issues in #1491.

I try to summarize the situation:

• The import statements in the generated python code depend only on the import statements in the .proto files, which means that the structure of the .proto files necessarily has to be the same as the python package.

  This is a big issue for people wanting to use .proto definition in python packages that are not directly related or have no control over the .proto sources.

• There currently are two hacky solutions for this issue:

  1. Replacing the import statements in the generated python code, e.g. using sed. This was actively discouraged by the protobuf maintainers, as generated code is no supposed to be modified. This is a reasonable stance in general.
  2. Messing with python's import path, e.g. via sys.path.insert(0, os.path.dirname(__file__)), sys.path.append(os.path.dirname(__file___)) or similar.

  This – in python terms – is the more hacky approach and create naming conflicts with other packages / stdlib files. E.g. imagine a package that has a random.py module. If this module would do the above, the stdlib random module is shadowed.

Actual solutions for this issue on the protoc level were rejected, e.g. in #7470.

• The only current way to get a working python package, is to directly have the .proto files in the correct structure.

  I created a minimal example here: https://github.com/maxnoe/python-proto-package/

  The structure is:

  .
  ├── protos
  │   └── prototest
  │       ├── bar.proto
  │       └── foo.proto
  ├── pyproject.toml
  ├── README.md
  ├── setup.py
  └── src
      └── prototest
          └── __init__.py
  
  

  calling protoc like protoc -I protos --python_out = src works in this case and creates proper import statements.

  While this works, it is not a solution for projects that need to include external proto definitions beyond their control.