Support for non-data manifest updates

[gasinvein]

flatpak-external-data-checker could potentially automate tasks other than external data version bumps (despite the name). Example use-cases (as of now):

• Update SDK/runtime branch
• Fix/optimize permissions (e.g. flathub/flathub#2180)
• Update git submodules in the manifest repo (#188)

We need some sort of abstract enough internal API for that, so this functionality could be easily extended in future as new update tasks appear. Any ideas?

[wjt]

@barthalion how would this overlap with your plan to write a linter? Maybe this tool should subsume that plan.

[barthalion]

I don't think it would, at least as long as the primary focus of fedc is to perform updates or migrations like described above. The idea for linter is to automate review process, as on multiple times we've merged apps with some outstanding issues, and also allow app maintainers/submitters to fix these issues before the actual submission.

[vchernin]

Not sure how the implementation should work, but one thing I realized is checking runtimes would probably need to be done with care.

If f-e-d-c runs and checks all repos for old runtimes, it would open a lot of PRs all at once. This would be very noticeable when a new runtime is released. This would probably overwhelm the build infrastructure. So perhaps f-e-d-c should pick 1 repo to open a PR in every time it runs (arbitrary choice). So in a day it would check and try to update 24 repo runtimes. If it can’t find any repos to update runtimes in well that’s a good thing.

f-e-d-c already seems to have a mechanism to "tell" if it’s an opened a PR or not so I guess that would help here? It would get through all repos with old runtimes within a few weeks/months maybe which I think is reasonable.

It also might be a good idea to have an opt out mechanism. Some repos for whatever reason might not want f-e-d-c often PRing about runtimes, permissions or submodules.

[vchernin]

FYI I started working on this, at least the submodule checker seems reasonably straightforward (including only updating a submodule if a referenced Flatpak module actually changed).

I am unsure on what to call the new checkers though. Right now I am using a placeholder "nonexternaldata" so the submodule checker is a "nonexternaldatachecker". Any better ideas for the name of this new group of checkers (the group including the runtime and submodule checker)?

Also, is an updating tool the right place for permission migrations? IMO it would be better to e.g. warn about gvfs holes in a linter, with suggestions for a fix depending on the problem. With the gvfs migration https://github.com/flathub/flathub/issues/2180, the correct change seems to depend on the app. Are there other examples of permission migrations where f-e-d-c could always correctly make a change?

OTOH, adding migrations here would be easier than writing a whole new tool.

[plata]

If f-e-d-c runs and checks all repos for old runtimes, it would open a lot of PRs all at once. This would be very noticeable when a new runtime is released. This would probably overwhelm the build infrastructure. So perhaps f-e-d-c should pick 1 repo to open a PR in every time it runs (arbitrary choice). So in a day it would check and try to update 24 repo runtimes. If it can’t find any repos to update runtimes in well that’s a good thing.

I'm not sure that this is the best approach. If there is an issue on the build infrastructure site, I believe that also the build infrastructure should handle this (e.g. by throttling/queuing the builds).

[vchernin]

I believe that also the build infrastructure should handle this (e.g. by throttling/queuing the builds).

Well, buildbot may be able to queue everything without breaking. But then you have a considerable mass of builds queued, which are competing with user-triggered builds. Maybe a solution could be to always prioritize builds triggered by users over bot builds, but I don’t know if buildbot supports that.

[Justinzobel]

Buildbot (or whatever handles build queue) could potentially be modified to understand two queues. One for regular builds and one for lower priority updates. If the regular build queue is empty, pick up a low-priority build. Fedora's Copr infrastructure has this, and it works quite well.