GitHub Action to purge old dashboard results.

[markcmiller86]

Is your feature request related to a problem?

If the Git repo backing up our test dashboard gets too big (e.g. >1GB), the web page will stop regenerating. You can see this when its happening by looking at the repo's Actions tab and look for pages build and deployment actions that have recently failed.

We should add a GitHub action that runs once a week or so and purges old test results.

[markcmiller86]

A Failing build due to size looks something like this...

[markcmiller86]

@brugger1 I've been thinking about this a bit more and have some other thoughts.

• The basic problem is that a gh-pages site starts with a git clone operation (I am pretty sure it clones the whole repository).
• LFS'ing the images might work ok. It would certainly make the clone happen faster. We can use githubs raw-content URLs to actual images in the site pages. That would probably work but it'd be a bit clumsy.
• We could split the repo into the basic results repo (images and text files) and the page-gen repo and just ensure the site gen properly builds the correct url/links to the basic results repo's raw content.
• Don't site gen at all. Instead of creating html indexing and navigation to the data, just create GFM (GitHub Flavored Markdown) indexing and navigation and then browse the stuff directly on github as markdown. So, instead of bunch of .html files pointing to other .html files and finally to images and text results, we'd have a bunch of .md files doing the same thing. We basically do all the site gen work ourselves anyways when we produce all that static html content. All gh-pages is buying us is given a URL to browse that html content natively.

None of this avoids the possible need to periodically purge old test results. But, LFS'ing or separating into two repos does mean git clone (for the site-gen step) would be a) way fast and b) always very small. The last option means git clone would never have to happen to begin with. But, it also means you need to go through GitHub.com to browse our dashboard instead of as raw-html. Some browsers have .md extensions now too.

[markcmiller86]

@brugger1 here is an example of an html-only dashboard that uses GitHub's raw user content URL linking to link to images in another repo.

• repository: https://github.com/visit-dav/dashboard-lite
• published site: https://visit-dav.github.io/dashboard-lite/databases_silo

I used just one example, silo.py and patched links for just the first couple of failed results. But, I think its enough to get the idea.

[markcmiller86]

Some new observations to report here.

• First, the dashboard repo is > 8 Gigs in size at this point. The 1Gigabyte limit referenced above is for the generated site, not the repo.
• Next, if we choose NOT to do any site-gen at all (e.g. no gh-pages site) and instead just browse via markdown, we loose some functionality like sorting tables by columns. I don't think we want to do that. So, I conclude we should continue to generate a real site.
• I think we can keep simple with one repo and all images LFS'd and have the genereated html reference the raw images so that the repo itself will never be very big. The git-lfs end of things would get really, really big, however.