v2 / experimental Go APIs

[anderseknert]

For the past few years, Regal has often pushed the limits of OPA's Go APIs, and helped improve them where possible. There are however APIs where the design / interface of the API itself impose limits we can't realistically change without also breaking the contract of those APIs.

Before providing some examples of this, I want to emphasize that this is almost never due to our APIs being poorly designed. But simply a consequence of some of those APIs being quite old by now, and much having happened in both Go and OPA since they were published. I'm not going to list them all here, but just to provide a few examples from APIs I have worked on recently:

• Frequent use of any where generic types would now be a much more appropriate option (and much more efficient compared to boxing concrete types to any)
• Bundle data expected to be provided in the form of map[string]any even though this will be nothing but an expensive intermediate format when recent storage implementations, like in-mem AST, are used
• File filtering functions demanding to be passed a fs.FileInfo argument, preventing the use of the much more efficient filepath.WalkDir function (Go 1.16+) which instead passes a fs.DirEntry for filtering (avoids stating every file visited)
• AST collection types with no consistent collection API, and each having public methods
• A storage API which requires refs to be converted to storage.Paths, only to then need to convert those back to refs for AST store lookups
• And so on...

My suggestion is that we introduce a new v2 or exp directory where we can work on the design for what should eventually become new APIs. These APIs should not be internal, but they will also come with no guarantees of stability, and can change from one version to another without notice. This to allow external cutting edge projects like Regal to test out new API versions without resorting to hacks. Naturally, we'll need to be very clear in communicating the rules for depending on these APIs.

The model that Go itself has used for its v2 APIs seems like a pretty good one to me. Meaning how they have the v1 APIs stick around "forever", but also have most of those implementations point to v2 APIs where possible, making them more or less facades for the newer APIs, and the benefits they commonly bring with them.

Needless to say, this should have no impact whatsoever on users of the OPA binaries, or Go API integrations making use of the v1 APIs.

[srenatus]

this should have no impact whatsoever on users of the OPA binaries

If it's more efficient, then I would think opa eval should make use of it, eventually, too. But I suppose you meant that the CLI itself wouldn't change, which is of course possible.

This is a big (and incomplete) list, so let's tackle this piece by piece...

[anderseknert]

Indeed, those users may notice a positive performance impact in some cases. I can add to the list for sure, but it’ll probably never be complete, as we find new things we want to do, or new features are introduced in Go. But if you all approve of the idea, I’d be happy to get the ball rolling on this one, and we can take it from there.

Probably best to start small. Perhaps with the file loader / filtering?

[srenatus]

If we're calling Santa, I was a very nice person this year and I want to have first-class fs.FS support in all rego.* methods.

[stale[bot]]

This issue has been automatically marked as inactive because it has not had any activity in the last 30 days. Although currently inactive, the issue could still be considered and actively worked on in the future. More details about the use-case this issue attempts to address, the value provided by completing it or possible solutions to resolve it would help to prioritize the issue.