guardian
Chromatic usage
All numbers from this script run against the CSV from chromatic.
We are currently using a lot of Chromatic, this issue is being tracked to see if there are any patterns we can find in the usage data to help us optimise this, or decide that it is just an operating cost we need to have.
We have enabled TurboSnap in August, and have seen significant amounts of skipped snapshots:
{
"0": {
"run": 281650,
"skipped": 0,
"dependabot": 17127,
"runOn": {},
"firstRuns": 74032,
"reRuns": 207618,
"summary": {
"total": 281650,
"run": "100%",
"skipped": "0%",
"dependabot": "6%",
"runSummary": { "firstRuns": "26%", "percentReRuns": "74%" }
}
},
"1": {
"run": 592987,
"skipped": 0,
"dependabot": 81870,
"runOn": {},
"firstRuns": 157288,
"reRuns": 435699,
"summary": {
"total": 592987,
"run": "100%",
"skipped": "0%",
"dependabot": "14%",
"runSummary": { "firstRuns": "27%", "percentReRuns": "73%" }
}
},
"2": {
"run": 676267,
"skipped": 0,
"dependabot": 81542,
"runOn": {},
"firstRuns": 190125,
"reRuns": 486142,
"summary": {
"total": 676267,
"run": "100%",
"skipped": "0%",
"dependabot": "12%",
"runSummary": { "firstRuns": "28%", "percentReRuns": "72%" }
}
},
"3": {
"run": 604545,
"skipped": 2825,
"dependabot": 43505,
"runOn": {},
"firstRuns": 161781,
"reRuns": 442764,
"summary": {
"total": 607370,
"run": "100%",
"skipped": "0%",
"dependabot": "7%",
"runSummary": { "firstRuns": "27%", "percentReRuns": "73%" }
}
},
"4": {
"run": 1039123,
"skipped": 0,
"dependabot": 99918,
"runOn": {},
"firstRuns": 313862,
"reRuns": 725261,
"summary": {
"total": 1039123,
"run": "100%",
"skipped": "0%",
"dependabot": "10%",
"runSummary": { "firstRuns": "30%", "percentReRuns": "70%" }
}
},
"5": {
"run": 796172,
"skipped": 0,
"dependabot": 85251,
"runOn": {},
"firstRuns": 228715,
"reRuns": 567457,
"summary": {
"total": 796172,
"run": "100%",
"skipped": "0%",
"dependabot": "11%",
"runSummary": { "firstRuns": "29%", "percentReRuns": "71%" }
}
},
"6": {
"run": 787009,
"skipped": 316044,
"dependabot": 33551,
"runOn": {},
"firstRuns": 212145,
"reRuns": 574864,
"summary": {
"total": 1103053,
"run": "71%",
"skipped": "29%",
"dependabot": "3%",
"runSummary": { "firstRuns": "19%", "percentReRuns": "52%" }
}
},
"7": {
"run": 913883,
"skipped": 965833,
"dependabot": 98512,
"runOn": {},
"firstRuns": 241274,
"reRuns": 672609,
"summary": {
"total": 1879716,
"run": "49%",
"skipped": "51%",
"dependabot": "5%",
"runSummary": { "firstRuns": "13%", "percentReRuns": "36%" }
}
},
"8": {
"run": 221870,
"skipped": 271850,
"dependabot": 27409,
"runOn": {},
"firstRuns": 77211,
"reRuns": 144659,
"summary": {
"total": 493720,
"run": "45%",
"skipped": "55%",
"dependabot": "6%",
"runSummary": { "firstRuns": "16%", "percentReRuns": "29%" }
}
},
"9": {
"skipped": 0,
"run": 0,
"dependabot": 0,
"runOn": {},
"firstRuns": 0,
"reRuns": 0,
"summary": {}
},
"10": {
"skipped": 0,
"run": 0,
"dependabot": 0,
"runOn": {},
"firstRuns": 0,
"reRuns": 0,
"summary": {}
},
"11": {
"skipped": 0,
"run": 0,
"dependabot": 0,
"runOn": {},
"firstRuns": 0,
"reRuns": 0,
"summary": {}
}
}
While we are seeing an >50% increase in the snapshots skipped, we are still seeing a massive increase in the snapshots created. This should be investigated (have we added loads of stories?).
We analysed to see how many times tests are run against a branch
i.e. { [runCount]: branchCount } - There were branchCount branches with runCount amount of runs.
e.g. There were 1 branches with 1936 runs (this is main so makes sense)
{
"1": 1012,
"2": 481,
"3": 271,
"4": 156,
"5": 101,
"6": 89,
"7": 43,
"8": 49,
"9": 28,
"10": 31,
"11": 25,
"12": 16,
"13": 12,
"14": 12,
"15": 9,
"16": 5,
"17": 6,
"18": 5,
"19": 3,
"20": 1,
"21": 5,
"22": 8,
"23": 1,
"24": 2,
"25": 4,
"26": 3,
"27": 2,
"28": 1,
"31": 2,
"35": 1,
"38": 1,
"44": 2,
"45": 1,
"46": 1,
"59": 1,
"1936": 1
}
Grouped
{ "1": 1012, "2-5": 1009, "6-10": 240, "11-50": 128, "51-100": 1, "100+": 1 }
Which leaves us at top heavy with suits being run 1-4 times which seems resonable.
Going off of this, it seems to be running on every push to the branch: https://github.com/guardian/dotcom-rendering/blob/be06d6c095db950d9a0542e814ec058916c3b9a7/.github/workflows/chromatic.yml#L3
In Identity we made a change ^1 so that we only run Chromatic when a re-review is requested on the PR. This allowed us to selectively re-run during development only when needed, as opposed to continually as we pushed.
Perhaps a similar workflow adaptation could be considered here?
@ashishpuliyel
Why would a branch have hundreds of runs against it?
There is only 1 branch with 100 runs against it, which is main. The next runner up is 59.
@ob6160 that was something we explored, but we thought it broke our workflow as we might forget to re-review as we do sometimes make requested changes, but do not re-request a review if those changes were small enough.
We had something similar where we ran it only on PR, which didn't see much change. That's why I was running the stats on seeing if there were a lot of branches with a lot of runs against them, which it doesn't seem so...
EDIT: I've added the breakdown of the run percent, which actually looks like most tests are run on 2+ pushes to a branch. This could be something to look at. We could also look at whether these branches have PRs against, but I think they might.
I still think, looking at the numbers, Turbosnap seems to be the most effective, and maybe make some workflow change on dependabot changes?
This is being closed off and we're going to choose a contract with them, DevX is leading on this.