Utilize Diffy for Visual Regression Testing

[quicksketch]

The owner of Diffy, a visual regression testing service, has offered their services pro-bono to the Backdrop project. @stpaultim, @izmeez, and myself (@quicksketch) met with Yuriy from Diffy today to discuss it.

This issue is to coordinate the evaluation and setup process.

[ygerasimov]

As agreed on the call we will set up Diffy for the main website https://backdropcms.org/. It was great talking to you all today.

[stpaultim]

We have an account set up and have configured it to take snapshots of beta.backdropcms.org and backdropcms.org and compare them.

This is a pretty basic implementation of Diffy, but a good place to begin.

We'll talk more about this at our user group meeting, next Thursday.

[izmeez]

I don't think I was on the initial call. But, this does sound interesting and potentially useful. I'll make an effort to attend the meeting.

[ygerasimov]

@stpaultim I see that staging environment miss images. Can we set up sync of the images from prod or maybe even better to have stage_file_proxy. Otherwise Diffy will show a lot of changes because of the missing images.

[stpaultim]

@ygerasimov - Yes, we'll look at that ASAP. I noticed this as well and am aware that the two sites are quite out of sync right now. So, we need to refresh the database on Beta.

Baby steps...

:-)

Atten: @bugfolder

[bugfolder]

So, we need to refresh the database on Beta.

It's a little more than just that, I think. Since the dbs of Beta (site) depend on the code in beta (the branch) and the files on beta (the server), the following all should be done (preferably at one time):

• Update beta branch of code from main branch
• Backup and copy files from b.org to beta.b.org
• Copy config files from live-active on b.org to beta-active on beta.b.org
• Backup the Backdrop db on b.org, import to Backdrop db on beta.b.org
• Backup the CiviCRM db on b.org, import to CiviCRM db on beta.b.org
• Fix any permissions that didn't end up with the right owner or group

Currently, b.org has a bunch of available updates: Backdrop 1.26.2, which just came out, of course, but also 9 contrib modules, plus CiviCRM that also need to be updated to the latest versions. It's not strictly necessary to update those before mirroring from b.org, but if the goal is to have Beta as close as possible to b.org (at least for a little while; the divergence starts almost immediately since b.org gets several spam account creations per day), it might be desirable to do all the b.org updates first, then carry out the mirroring process.

Here's the release checklist for 1.26.2: https://github.com/backdrop/backdrop-issues/issues/6315

[izmeez]

I may not yet be dialed into the objectives: what is it that we want to differentiate with the help of diffy? Is the primary objective to confirm the code and configuration or to have a clone?

Since the database contains content and possibly some config, it would be considered as required to copy and sync. The config files and database are combined with the demo module or can be done with equivalent commands.

Files can be done through copy or a module to access the files remotely, as one can do between production and staging sites. Likewise, the CiviCRM database could be copied or accessed remotely.

Will remote access for these help focus what we are trying to differentiate?

[stpaultim]

@izmeez - Bugfolder had done this and knows what needs to be done, he is just reminding me what is involved (at least, that is what I understand).

Diffy is simply a visual regression tool that we're experimenting with and we decided that the first step towards experimenting with it, would be to try using it on BackdropCMS.org. What it does is take snapshots of about 20 pages on backdropcms.org and then we can push experimental changes to the beta site and it will take the same 20 snapshots and analysize them to see if there are any unexpected differences.

We're very much in expemental mode and after trying this, we can see if there are other ways we might use Diffy as a community. The developers of Diffy have offered us free access for community use. At this point, we're simply testing Diffy to see how useful it could be and how we might best take advantage of this generous offer.

https://diffy.website/

@bugfolder - I've avoided getting involved in this before, but will give it a try. Maybe, I need to start by making sure I have a working local copy of b.org. I'll let you know if I have questions.

Does the database need to be sanitized before importing into beta site?

[bugfolder]

Does the database need to be sanitized before importing into beta site?

No, beta site contains real user data (hence the hiding of it behind a password).

[klonos]

Hey @ygerasimov 👋🏼 ...nice to see you here (do you remember me from Salsa Digital?). Thank you for offering your services pro-bono to our project 🙏🏼

@bugfolder are we using https://github.com/backdrop-contrib/stage_file_proxy on the beta site? We should! It would reduce the tasks that we need to be manually doing.

[klonos]

...also, can I please be added as a member to the team/organization in the Backdrop Diffy account?

[ygerasimov]

@klonos 👋🏼 great to see you here! Please advise your email address (drop me an email to [email protected]) and I'll send you the invite.