Renovate support for buf.lock

[nightlyone]

I just filed a feature request to manage buf.lock updates using renovate at #renovatebot/renovate/issues/17802

Maybe you want to track that work and chime in to support the implementation.

That seems quite important to me, because it looks like in the buf ecosystem buf module versions can disappear after a while leading to failed builds of working software.

[viceice]

Is there any official API for the BSR? That is required for renovate to find possible updates.

[viceice]

• https://github.com/bufbuild/buf/issues/1022
• https://github.com/renovatebot/renovate/issues/17802

[bufdev]

That seems quite important to me, because it looks like in the buf ecosystem buf module versions can disappear after a while leading to failed builds of working software.

We made a one-time change during the beta of the BSR as outlined in https://github.com/bufbuild/buf/issues/1380. Once the BSR is marked stable (extremely soon), this will no longer be a concern.

[bufdev]

Is there any official API for the BSR? That is required for renovate to find possible updates.

There's no official API, but https://github.com/bufbuild/buf/blob/main/proto/buf/alpha/registry/v1alpha1/resolve.proto#L33 would be what you want. The API is technically in alpha, but we'll have to support this endpoint going forward for the Buf CLI, which is in v1.

[Vinzius]

Hi!

I investigated/coded a bit the subject last week. I guess several people had the idea following the recent BSR breaking change (e.g. googleapis).

I'm laying down here what I learned from trying to implement from scratch a datasource/manager for Buf. Not sure that's the best way to go, but it allowed me to better understand how RenovateBot works.

Renovate mostly works with three things:

• Manager : read/write the deps (so the buf.yaml + buf.lock).
  • Would need a custom manager for Buf.
  • Read/write would likely be achieved through regex (like most managers). Using something like js-yml package would be easier, but some data could be lost (e.g. comments).
  • Need to handle different "update types", and it's not straightforward..
• Datasource : check the latest releases (& metadata like tags) for a dependency.
  • Would need a custom datasource for Buf.
  • Could use REST calls (like the BSR website does?). For instance getting repository info https://api.buf.build/buf.alpha.registry.v1alpha1.RepositoryService/GetRepositoryByFullName or repository commits https://api.buf.build/buf.alpha.registry.v1alpha1.RepositoryCommitService/ListRepositoryCommitsByReference with "json post calls". It works (I verified). But maybe not "stable".
  • Unlikely that RenovateBot would include buf CLI to simplify updates.
• Versioning : format of a release. The most well know is semver (e.g. v2.24.0).
  • No need to create a "buf" one if we could reuse another one (e.g. use git versioning).
  • But it seems to be used to sort versions and decide which one is the latest. In our case it would be with the timestamp?

I opened a branch while testing. Code isn't prod ready at all, I tried to have an e2e thing working (that you can launch from Renovate code with yarn start @you/testing-repo). Almost working, but not quite yet :) (mostly due to the versioning issue, and another one I didn't get).

How I see the next steps:

• Discuss details in this thread.
• Determine the overall logic (and so see if we have the API route to work). I would write a draft.
• Buf Team could fill the survey that RenovateBot Team requires to decide if a new manager should be accepted. And add it in the current thread.

What do you think?

[bourquep]

Is anyone working on this? If not, I might give it a try.

[nicksnyder]

As a point of reference, we recently investigated adding support to Dependabot and concluded that there wasn't a viable path to making something useful.

The key challenge is that it isn't as simple as updating buf.yaml and buf.lock. To make a PR mergeable you also need to regenerate code in the PR, which tends to be project specific and is fundamentally arbitrary code execution. This means that the flow would need to looks something like:

1. Renovate opens a PR to update the buf.yaml and/or buf.lock
2. Somehow have a separate project defined workflow automatically run in response to the renovate PR to generate code on the renovate branch.

An alternative solution is to use Generated SDKs that the Buf Schema Registry provides. These look like normal packages in your language's ecosystem and will get updated in the same way as your other language dependencies (aka Renovate works with them if you already have that configured).

[bourquep]

In our case, the use-case is pretty simple: we have a bunch of repositories that define their own Buf modules, and there are interdependencies between those modules. When module B changes, we need to update the lock file of module A for example. We push to the BSR during CI, so it's not Renovate's job to do that.