Allow SPADS tracking of rapid tags for versions

[p2004a]

Right now hosting preset's modName only accepts the regexp of the mod name, and always uses the lexicographically newest one. It would be useful to be able to instead track rapid tag that could be repointed at an older version, e.g. due to rollback etc.

Another use case is to track some more stable tag, but allow to have newer versions installed on the server that are selectable by privileged users via !hset modName+!rehost.

As a workaround it could be implemented externally by some separate application updating modName in settings, but I think it would be a nice thing to have built in.

[Yaribz]

I'm not sure I fully understand the request. Do you want SPADS to resolve the rapid tag from the local rapid repository (i.e. an external process or a plugin would be responsible for updating the local rapid repository independently), or do you want it to resolve the tag from an online rapid repository ? (and auto-download it using pr-downloader ?)

[p2004a]

The former: resolve tag from local rapid repository.

[Yaribz]

In this case I see a few points which might need clarification:

What if several local rapid repositories are available (for instance repos-cdn.beyondallreason.dev and repos.beyondallreason.dev) ? Should it take the most recent versions.gz file according to modif time ? Should it follow some rapid resolution order setting and only check the first matching, or allow a configurable unique specific local repository ?

Also, what if the configured rapid tag points to a version which is not available in local rapid repository ? For example, SPADS is configured to track rapid tag byar:testing, but the latest run of pr-downloader was with parameter byar:stable. Thus pr-downloader updated the local rapid repository to retrieve the game packages corresponding to the tag byar:stable, but it may also have updated the local versions cache for the byar:test tag, which would now point to a version which would not be available locally.

[p2004a]

What if several local rapid repositories are available

Engine has explicit setting for specifying priority that for this reason https://github.com/beyond-all-reason/spring/blob/cb7d17009cf3f31793e72cd3356bfa752a897dc2/rts/System/FileSystem/RapidHandler.cpp#L91-L137.

Also, what if the configured rapid tag points to a version which is not available in local rapid repository?

Good point, fail in the same way as if somebody sets modName to a not-available version?

For the use case I have in mind, we could always just call pr-downloader under archive lock with --download-game byar:test --download-game $some_other_verion, so that we don't run into this issue.

[Yaribz]

Engine has explicit setting for specifying priority that for this reason https://github.com/beyond-all-reason/spring/blob/cb7d17009cf3f31793e72cd3356bfa752a897dc2/rts/System/FileSystem/RapidHandler.cpp#L91-L137.

Yes I know Recoil has this setting (but Spring doesn't...). Tbh I would be tempted to just use the repository containing the most recently modified versions.gz file.

Good point, fail in the same way as if somebody sets modName to a not-available version?

If, at the time the modName setting value is configured by an administrator to some rapid tag, the corresponding version isn't available in local rapid repository, then indeed it makes perfect sense that SPADS startup fails with some "unavailable game version" error (similarly to existing behavior). However I find it very counter intuitive that just by updating the local rapid repository afterward (downloading new versions without removing anything), one can end up breaking an existing SPADS configuration. I know it is possible to avoid this problem by gathering all the required pr-downloader actions in one lock-protected "transaction", but still this is error prone as one can easily forget it...

The more I think about it, the more I think I should just extend the existing BarAutoUpdate plugin, so that it would also auto-set the modName setting to the resolved rapid tag set in the plugin configuration. This would allow dynamic game version support based on rapid tag tracking, without necessarily using the latest one in lexicographic order. As it is the plugin itself which is performing the game archives updates, based on the same rapid tag as the one used to update the modName setting automatically, consistency would be guaranteed and it would be impossible to forget anything.

The plugin already handles all the required locking actions automatically, and all its functionalities can be enabled separately. So the plugin could be used to do exactly what is described here and nothing else if configured so.

[p2004a]

Ok, so I hear that doing it in plugin/separate software is preferred solution to do this because it's too tricky to make the functionality reliable enough for core SPADS and the description makes sense for me.

Thank you @Yaribz for discussion!

[Yaribz]

Ok, so I hear that doing it in plugin/separate software is preferred solution to do this because it's too tricky to make the functionality reliable enough for core SPADS and the description makes sense for me.

To be specific, I would say that implementing the rapid tag tracking functionality in a component different from the one performing the rapid repository update can indeed be tricky to maintain reliably from administrators point of view, especially when multiple rapid tags are involved.

In SPADS case, the rapid repository update functionality is currently implemented by a plugin, so I'm inclined to implement the rapid tag tracking functionality in this plugin as well, to ensure consistency (i.e. the tag selected for hosting is also selected automatically for rapid updates).

This is not entirely decided though, as I didn't have time to really think about it yet. For instance, there might be problems when several instances are hosting different rapid tag simultaneously, because with current plugin implementation only one instance will perform the update of the shared rapid repository, so only one tag would be automatically updated...

Anyway, I think I will implement this feature one way or another, so for now I reopen the issue to keep track of it.

Thank you @Yaribz for discussion!

Thank you for bringing this point, as I didn't even realize there were such use cases that weren't handled correctly by current plugin implementation.

[Yaribz]

FYI I found some time to think about it a little more, and I changed my mind.

My plan, for now, is:

1. Extend the BarAutoUpdate plugin to support multiple rapid tags simultaneously for the game auto-update functionality.
2. Add support for rapid tag as modName hosting setting value in SPADS core (with proper caveats in doc, and maybe with automatic fallback to last known version available locally matching this tag)
3. Add a warning in the BarAutoUpdate plugin when the modName hosting setting of SPADS core is set to a rapid tag that is not in the list of the auto-updated game tags.

[Yaribz]

Starting with version 0.13.44, SPADS now supports tracking of (local) rapid tags for versions.

Usage can be found in doc.

My plan, for now, is:

1. Extend the BarAutoUpdate plugin to support multiple rapid tags simultaneously for the game auto-update functionality.

done in commit update BarAutoUpdate plugin (v0.4)

2. Add support for rapid tag as modName hosting setting value in SPADS core (with proper caveats in doc, and maybe with automatic fallback to last known version available locally matching this tag)

done in commit add support for rapid tags in "modName" setting

3. Add a warning in the BarAutoUpdate plugin when the modName hosting setting of SPADS core is set to a rapid tag that is not in the list of the auto-updated game tags.

done in commit update BarAutoUpdate plugin (v0.5)