GAT - Completely automated recordings

[hexylena]

We are not here yet, but we're getting closer.

Support Infrastructure

• [x] #124 Using Diffs universally to extract, working.
• [x] Automated extraction of commits into git-gat repository https://github.com/hexylena/git-gat
  • [ ] Testing: #127 We need to test the roles with molecule (Help wanted, see issue for more details on molecule)
  • [ ] Testing: We need to do some sort of smoke testing with the deployed galaxy, check that features work, etc. (Help wanted)
• [x] #125 Commands that are run are annotated
• [ ] #126 For user friendliness, the GTN needs to make diffs easier to copy.

Open Issues from GAT21v1

See "Role Changes" https://github.com/galaxyproject/training-material/issues/2274 (Help wanted: ansible)

Building completely automated videos

• Working Proof of Concept https://github.com/hexylena/auto-gat
• Terminal Recordings
  • [x] Commits from git-gat are loaded into a github worker
  • [x] Commands that need to be run are identified
  • [x] These commands are run, output recorded
  • [x] Video cleaned up for long pauses
  • [x] Uploaded to asciinema https://asciinema.org/~hexylena
  • [x] Rendered to mp4 https://github.com/opcode-eu-org-libs/asciicast2movie
• GTN Screencaptures
  • [x] Annotated where we want screenshots/videos. (cvmfs tuto only)
  • [x] convert diffs into 'file editing' (fake vim or so? we just need a script that goes step by step through the diff and shows the edited screens one by one, it can be python but needs to output correct width/etc.) Blocker from @Slugger70, thanks! Maybe https://github.com/tdryer/editor works for us
  • [x] Need to render GTN + scroll to specific position
• Galaxy/web screencaptures
  • [ ] Need to record complex web interactions (e.g. visit galaxy, click login, register, click 'assembly' in tools menu, click bwa, click the ref genome selector)
• Muxing
  • [x] Stitch above mp4s with audio from polly
  • [ ] Stitch together for resulting video.

This would simply provide the 'base level' course materials (polly is an improvement over slides after all.) If you run this again, of course you would have humans produce some of the videos, but it would provide video instruction when time/money for making the videos isn't there.

[hexylena]

Tested https://github.com/qawolf/playwright-video during a meeting I was not paying attention to and it seems pretty ok? maybe we can make that work.

not the prettiest thing but that's because I'm copying and pasting with zero knowledge of this area.

const { chromium } = require('playwright');
const { saveVideo } = require('playwright-video');

(async () => {
  const browser = await chromium.launch();
  const context = await browser.newContext();
  const page = await context.newPage();

  await saveVideo(page, 'video.mp4');
  await page.goto('https://usegalaxy.eu');
  await page.waitForLoadState('networkidle');
  await page.waitForTimeout(5000);
  await page.click('#tool-panel-upload-button');
  await page.waitForTimeout(5000);

  await browser.close();
})();

Combined with 42f1eb8 (#2371) we'll have a script and read that script and turn it into page.clicks, annotated inline. (TODO: better syntax?)

> I'll click on Analyze Data to just to reload the page.
> await page.goto('https://gat-0.eu.training.galaxyproject.eu');
>
> I'll go back to Mapping and load bwa-mem. And then suddenly, instead of
> having no options available, you can see here we've got the Bee genome.

await page.click('find mapping SOMEHOW')
await page.goto('https://gat-0.eu.training.galaxyproject.eu/root?tool_id=bwa-mem');

>
>  Now click on that. Oh look at that, there are lots and lots and lots of
>  available genomes now including: lots of human, mouse, rat, yeast, all sorts
>  of things. And in fact if you want to see the list of all the different
>  available genomes now, that we have available to us.
>  

await page.click('#s2id_field-uid-11_select');
(can we say fragile? maybe we pin the vesion of bwa-mem.)

>  If you go to admin. We go to data tables over here. You can see here that we
>  have um a couple of data tables for managed and for all fasta. So if we
>  click on that one, you can see that we have a lot of genomes available now
>  in the all fasta data table that Galaxy can get access to. If we go back to
>  the data tables again, and go down to bwa indexes or bwa mem indexes here.
>  You can see we have access to a lot of pre-built indexes for bwa for all of
>  these different genomes. That is pretty powerful.


await page.goto('https://gat-0.eu.training.galaxyproject.eu/admin');
await page.click('admin-link-data-tables');
? no clue how to find the first TR with text 'bwa_indexes' to click on

>
>  So what did that take us? Maybe 30 minutes? And uh suddenly our Galaxy
>  server has access to all the uh the data the reference data and the tool
>  indices that the community have built over a number of years and it's super
>  simple.

and we see something like:

https://www.youtube.com/watch?v=AtJ0R_FYIW4

[hexylena]

• https://youtu.be/PP6Dpq7uBTk example viewing the GTN and discussing what we'll do
• https://youtu.be/_As374KPLTo moving to the terminal, running some commands, and discussing.

[hexylena]

https://youtu.be/Ayb6r2J7l1I Improved version of terminal recording (and some debugging statements), thanks to suggestion from @Slugger70 that we really needed to show 'interactive' editing rather than just diffs.

introducing a new, terribly named bit of code https://github.com/hexylena/condensed to solve that issue.