Split this repo?

[timburks]

trafficstars

Our recent efforts to configure a multi-module repository did not go well, but the motivating issue remains -- the growing applications in the cmd directory add dependencies that aren't needed by consumers of the generated code in the openapiv2 and openapiv3 packages.

Splitting the repo seems like a more conservative approach to this problem, at least in the sense that it's less reliant on a somewhat exotic and less-understood Go feature that is at least lightly discouraged.

Possible reorganizations:

1. The first reorganization that I think of would be to move as much as possible to a new gnostic-tools repo, keeping the generated components in place. This avoids breaking users of these components, but would force users of the tools (like protoc-gen-openapi to look elsewhere).
2. Another would be to create new repos containing the generated artifacts (essentially, the contents of openapiv2, openapiv3, and possibly discovery, either each in its own repo or all in one). These repos would be almost-entirely machine-generated from tools in the base gnostic repo, would have a very limited set of dependencies, and clients could switch to these at their convenience. These would be very similar to generated client libraries in the googleapis org. This also preserves the issues and PR history of the main repo.

Of these, I prefer the second. Are there others that we should consider?

@Jefftree @morphar WDYT?

[Jefftree]

/cc @apelisse @liggitt

Thanks for laying this out Tim.

I haven't used any of the gnostic tools so please take my suggestion with a grain of salt. I feel like the first approach would make it much easier for projects that import gnostic to upgrade. I went through the experience of upgrading from a googleapis/gnostic to google/gnostic version, and when there are multiple dependencies that import gnostic and all depend on each other, syncing their gnostic version upgrade and package rename is quite challenging (eg issue: https://github.com/kubernetes/kube-openapi/pull/285#issuecomment-1067422948).

It seems like both suggestions go down the route of splitting up the dependencies. In terms of timeline, how quickly would we be able to get approval for a new repo to split up the code?

[morphar]

Hi @timburks Thanks for asking :)

I'm not sure I understand option 1 entirely, what do you mean by: ... but would force users of the tools (like protoc-gen-openapi to look elsewhere).? Is the idea that e.g. protoc-gen-openapi would be moved to gnostic-tools?

That said, it seem like there already is an established pattern, that supports your second option. gnostic-grpc and gnostic-go-generator are already "dependent" on gnostic, but lives in their own repo. Though these are gnostic plugins, it kinda make sense to follow that pattern.

There is currently 3 very different uses of this repo.

1. gnostic the command / tool (living in root)
2. Usage of gnostic code for other commands (living in /cmd/)
3. Simple import of gnostic libs in other repos

It might make sense to have a clean repo for gnostic related code and repo(s) for tools that utilize code from the gnostic repo and currently lives in /cmd/.

Hope that helps.

[morphar]

Just one more thought:

I actually really like the idea of a one-stop project, that can handle everything. gnostic might not be there yet, but it could easily be a target for a v1.0. If everything is split into different projects, there is a risk that each project will have slightly different opinions on how things should work, which could break compatibility with other projects in subtle ways. That would be a shame in my humble opinion.

[liggitt]

Structurally, I agree that moving main package commands that bring additional dependencies out into their own repo(s) makes more sense. runnable commands are dependency magnets... I'd suspect that over time, you may even want to isolate them from each other.

Practically, from the perspective of a kubernetes maintainer, I don't have strong feelings if the openapiv2 / openapiv3 libraries/packages are made available with minimal dependencies (yaml, protobuf, etc) via another module. We have exactly two repos we would need to update, k8s.io/kube-openapi and k8s.io/kubernetes, and could manage that without too much trouble. I'm more interested in being able to quickly pick up library updates without the additional dependencies.

[timburks]

After further discussion with @Jefftree , I've created a new repo https://github.com/google/gnostic-models that is intended to be a shadow of the gnostic models that kubectl and similar tools are currently using. kubectl would be updated to use these (a test substitution built with no problems). Benefits to model users:

• minimal dependencies and easier security audits
• tight change control (although clients should still use tagged versions)
• less churn
• easier import into the "thirdparty" section of your company's monorepo.

The main repo would stay as-is with multiple child projects, which I think is consistent with @morphar's suggestion that this would help us maintain consistency among projects. Future efforts like https://github.com/google/gnostic-grpc might now be better done in the main repo.

As familiarity with and ease-of-use of the Go module system grows, gnostic may become a multi-module repo, but that wouldn't be an urgent need. Users of models could get lightweight, low-dependency distributions by using the gnostic-models repo. Tags in the gnostic-models repo would match tags in gnostic, though there would likely be some tags in gnostic that aren't reflected in gnostic-models.

I think we would want to automate updates to gnostic-models as much as possible and include tests in that repo as it matures.

[morphar]

Sounds good @timburks 👍