Indexing and distribution redesign

[amis92]

The problem

We're not scalable. Full stop.

The current design of our organization is not scalable to the increasing popularity, needs and update frequency. The system has been failing intermittently (#208) since almost the beginning, but it was good enough.

Now it's not.

The proposal

We need to decentralize this operation. Keep liability and maintenance at minimum. As of writing this post, BSData has 119 repositories. There already was a proposal to deal with the issue: #112 - it was done at 49 repositories.

This means that datafiles should not be BSData-org exclusive. We need to allow indexing of third-party repositories as well. So that a person can setup their own repository and in some semi-automated way add it to the index. This is one part.

The other is that #112 needs to be solved - and the recipe is easier than ever thanks to GitHub Actions. BSData will provide an action that will do all necessary operations so that the BSData indexer only needs to read some manifest file of that repository/release. Each repository will run action by itself, thus offloading the indexer.

The work

Of course, being a volunteer-run entity, there's close to no momentum. I'd love to gain some support from community in refining and implementing this proposal.

I'll be slowly working on that myself, if anyone is wanting to contribute, please do.

[amis92]

This work is in full tracked via https://github.com/orgs/BSData/projects/3

The current state is as follows:

https://github.com/BSData/gallery together with https://github.com/BSData/publish-catpkg provide a functional replacement for the appspot webapp backend.

• Gallery contains a list of registered repositories ("registry")
• Gallery runs a scheduled workflow to update details of registered repos and saves them as "index". Details include:
  • repository latest release (including catpkg.json asset)
  • repository details like "archived" status
• When "index" changes, a bsdata.catpkg-gallery.json asset is uploaded to a Gallery release. This file contains list of repositories in the format recognized by BattleScribe as a RepositorySource.
• Specific repositories in RepositorySource (catpkg-gallery.json) point to catpkg.json assets of the latest release of a given repository.
• catpkg.json release assets are uploaded (along with all other necessary files) by the publish-catpkg Action. It should be run as per README (on release publish and edit).
• catpkg.json format is recognized by BattleScribe as the RepositoryDetails which provides information about all datafiles - their URLs, names, and revisions. Based on revisions, the app chooses which files to download. Files are added to release assets by the publish-catpkg Action.

[SamwiseOfCat]

Hi @amis92 - sorry to bother you again. I followed your excellent guides and created a repo, release etc but the final step to package the release is failing for me. Ive tried running the publish-catpkg Action 3 times and it always seems to fail here -

Any ideas on why th catpkg is failing? Thanks :-)

[amis92]

Above will be tracked in https://github.com/BSData/publish-catpkg/issues/20

[amis92]

Closing as abandoned due to repo archival: #269