Expose code templates in ygen

[briantopping]

Hi all, just getting started with YGOT, thanks for your continued investment in this project over time! I was going to comment on #412 as it discusses consumers of the public APIs, but created an issue as I realized there are more data points I wanted to reference for clarity.

Our use case is to feed the generated structs for a YANG schema into Kubebuilder. The composition of these types would open up a lot of options for disaggregated control plane options! So far, generation using include_schema=false to a single file creates structs that can be manually aggregated into very usable Kubernetes custom resources.

In its parse, Kubebuilder captures object and field comments, field names and types, struct tags and other aspects to generate additional metadata. To generate a CRD, these are restructured into OpenAPI v3 schemas in a form that they call a "structural schema". It's all very specific to Kubernetes and Kubebuilder, but it's our top candidate at the moment for the developer UX of the overall solution.

What I had hoped to find in YGOT / ygen is the template definitions were somehow modifiable, but they are statically defined as private local variables.

A couple of solutions come to mind, the obvious one is to make access to these templates public and override the class, but that would make overrides brittle to future change. At the other extreme, it's not DRY to duplicate the entire generator except on a fork, which we'd like to avoid.

I am probably missing a lot of context here, so wanted to get feedback if @wenovus or @robshakir had time to consider it. Another concern is the tag setting code, which we need to add YAML serialization tags and the model description attributes, which aren't a part of the typical Go template context. The template contexts could be made larger, but at a cost of maintainer confusion for unused data. It would work if the IR exposed yang.Entry (even if the internal generators didn't use it).

If we were to generate a PR, what kinds of objectives might reviewers be looking for leading to a successful merge? Of course there are many options and we don't want to create a lot of work on your end – or end up with an unintentional fork on our end. There's no guarantees (as in life), just discovering options that are a good fit with general community constraints. Thanks!

[robshakir]

hi!

Apologies for the delay getting back to you. This sounds an interesting use case.

Our goal with the IR was to create something that exactly allowed this kind of use case to be implemented -- albeit we were thinking different kinds of output (e.g., that which is created by ygnmi, protobuf, or other languages). I think you fell into a hole in our thinking here, where you're still trying to create Go, but you want to create a different type of Go. Hmm :-)

Immediately, I don't like the idea of exposing the yang.Entry in the IR -- since this is somewhat what we're trying to shield users from (i.e., hide the complexities of YANG whilst allowing them to generate their own output).

I don't think making these tags (especially if they include the description) on-by-default is something that we want to do, since this will come at the cost of very large generated code. In production uses of this library, we know users already care about this kind of size of source code (definitely during development, since it makes the files unnavigable -- we've implemented splitting the schema up for this reason).

It seems like, for your use case, you'd be OK to call gogen.Generate which returns a structured set of contents that are essentially the generated code, which you can then figure out how to output to files other than the Structs field of that return type returns the output of writeGoStruct.

One thing we could do is allow you to provide a way to overload the function that is called in the writeGoStruct phase. This seems much more robust than just allowing you to override the templates, since they might change. It does, however, mean that you will need to implement some of the internal logic of writeGoStruct such as how to generate list fields, or unions.

The next level down that could be exposed is the set of input types to the templates. These would become a public API. We'd split writeGoStruct at this point to allow it to return the set of inputs, that then you could go ahead and generate into your own code. This would essentially give us something like:

type PreTemplateGoCode struct {
  StructDef *generatedGoStruct
  ListKeyStructs []*generatedGoMultiKeyListStruct
  OrderedMapStructs []*generatedOrderedMapStruct
  ListMethods []*generatedGoListMethod
  LeafGetter []*generatedLeafGetter
  LeafSetter []*generatedLeafSetter
  DefaultMethod *generatedDefaultMethod
}

With reasonable unit testing, this could be an OK API to maintain. We'd have to make some changes to these output types (e.g., goStructField would need a description field, as would generatedGoStruct) ==> and all would need to be made public.

Generally, my objective functions here are:

• avoid you having to fork gogen and hence maintain duplicate logic should we find a bug in our Go generation code.
• avoid having a if(specialKubernetesFlagIsSet) in the gogen code itself such that there's more cognitive overhead of reviewing gogen PRs.
• avoid reviewers of ygot having to understand the custom logic that you have, and similarly, avoid you having to wait for ygot reviewers to be able to change your code.

I think this one is complex enough that it's worth:

• having a design doc to suggest the approach.
• prototyping to check that our proposed approach works and "smells" OK.

If you're open to this, I'm happy to review proposals.

thanks, r.