api-interfaces DRAFT: Using the buf.build toolchain for generating language bindings

DRAFT: Using the buf.build toolchain for generating language bindings

Open johnsabath opened this issue 2 years ago • 1 comments

Improves UX of building and consuming api-interfaces

Leverages buf's remote builds to avoid having to install most system dependencies on the host machine
Generates python, go, and typescript web bindings that are valid installable dependencies into a gen directory
Generates correct python import paths, avoiding the need for import path hacks

There are no breaking changes to the gRPC interface as a result of this change, although some imports may need to be updated.

Changes

Generation Command

yarn generate

Python SDK

Before:

# setup.py
    install_requires=[
        'grpcio==1.48.1',
        'grpcio-tools==1.48.1',
        'protobuf==3.19.5'
    ],

# some python file that wants to use grpc
thisPath = pathlib.Path(__file__).parent.parent.resolve()
genPath = thisPath / "src/stability_sdk/interfaces/gooseai/generation"
tensPath = thisPath / "src/stability_sdk/interfaces/src/tensorizer/tensors"
assert genPath.exists()
assert tensPath.exists()

sys.path.extend([str(genPath), str(tensPath)])

import stability_sdk.interfaces.gooseai.generation.generation_pb2 as generation
import stability_sdk.interfaces.gooseai.generation.generation_pb2_grpc as generation_grpc

After:

# setup.py
    install_requires=[
        'grpcio==1.51.1',
        'grpcio-tools==1.51.1',
        'protobuf==4.21.6',
        'stability-api-interfaces @ git+https://github.com/stability-ai/api-interfaces.git@js/buf-spike#subdirectory=gen/proto/python',
    ],

# some python file that wants to use grpc
import stability_api_interfaces.gooseai.generation_pb2 as generation
import stability_api_interfaces.gooseai.generation_pb2_grpc as generation_grpc

Full Diff

Go

Before:

import (
  "github.com/stability-ai/api-interfaces/gooseai/engines"
  "github.com/stability-ai/api-interfaces/gooseai/generation"
  "github.com/stability-ai/api-interfaces/gooseai/dashboard"
)

After:

import (
  "github.com/stability-ai/api-interfaces/gen/proto/go/gooseai"
)

Testing status:

[x] Verified python bindings with Python SDK
- https://github.com/Stability-AI/stability-sdk/compare/main...js/buf-spike
- All tests pass
- CLI works locally
[ ] Verified python bindings with generator service
[ ] Verified go bindings with REST and GRPC APIs
- [x] REST commit: https://github.com/Stability-AI/api-bridge/commit/7305f4b3a8e9c3a985c9c9521e60c41275a39be9
[ ] Verified js bindings on JS SDK / DreamStudio
- Grrr, this is trickier because while npm and yarn do support installing a package from a git repo, they don't support installing from a subdirectory of a git repo. Can likely be worked around via a git submodule, but still annoying.