yari chore(scripts): add macro-usage-report

Summary

Adds a script that reports how often each macro is currently used in yari and in each locale.

Problem

We have a lot of macros, and we don't have an overview of how often each macro is used, especially since some macros are only called internally by other macros.

Solution

Implement a script that counts macro usages in mdn/content, mdn/translated-content and mdn-yari using rg (ripgrep).

Result

(Last updated on 2022-09-15 at 13:45 UTC.)

macro	yari	en-us	es	fr	ja	ko	pt-br	ru	zh-cn	zh-tw
AddonSidebar		671	22	627	193	36	32	43	111	29
AddonSidebarMain		1			1
APIListAlpha		1	1	1	1	1	1	1	1	1
APIRef		5358	522	1276	2361	583	428	734	1912	204
ARIARole		39	5	9	38	29	3	20	23	4
AvailableInWorkers		270	45	185	160	57	35	32	105	10
bug	1	220	86	133	218	44	70	98	128	39
CanvasSidebar		16	9	15	16	15	13	15	15	13
Compat	1	8555	1265	3746	4566	1480	1269	1753	3475	488
CompatChrome 🗑			3		1		1	6
CompatGeckoDesktop 🗑			1		1		5	2
CompatGeckoMobile 🗑			1		1		5	1
CompatibilityTable 🗑					1		10
CompatNo 🗑			1				6	2
CompatUnknown 🗑			1		1	1	7	1
CompatVersionUnknown 🗑			1				10	2
CSP		45	4	17	36	6	5	4	21	2
CSS_key_concepts 🗑			13				17	18
CSS_Ref		1	1	1	1	1	1	1	1	1
CSSAnimatedProperties		1		1	1	1			1
CSSInfo		466	207	442	463	120	92	142	265	26
CSSRef		959	432	893	893	282	220	329	580	60
CSSSyntax		454	260	561	552	157	125	201	353	32
cssxref	2	1576	522	820	1237	393	315	468	805	96
DefaultAPISidebar		433	26	61	179	50	28	58	98	16
Deprecated_Header		677	35	105	255	17	37	45	106	9
Deprecated_Inline		365	61	163	199	97	49	55	161	50
DeprecatedBadge	8
DeprecatedGeneric	2		4			2	5
DiscussionList 🗑			6				6	3
DOMAttributeMethods		13	6	12	13		3	4	12
DOMxRef	6	6263	647	1305	2976	880	631	1035	2300	338
EmbedGHLiveSample		190	45	112	130	34	26	48	76	11
EmbedInteractiveExample		1183	225	980	997	510	199	262	578	108
EmbedLiveSample		1540	395	1005	1153	375	257	443	788	101
EmbedYouTube		18	7	16	16	3	4	9	11	6
event 🗑			104	108	161	130	99	159	304	57
experimental_inline		322	62	128	127	84	82	138	134	41
ExperimentalBadge	9			2
Firefox_for_developers		104	5	29	98	1	4	12	21	4
FirefoxSidebar		143	23	44	69	2	9	18	31	19
fx_minversion_note		2	2	3	3	1	2	1	5	2
GamesSidebar		69	34	32	38	17	14	37	51	8
gecko			12	13	16	4	16	17	22	13
gecko_minversion_header 🗑			16				12	6
gecko_minversion_inline 🗑			23				22	27
geckoRelease 🗑	2		30	42	26	18	40	54	55	26
Glossary		1520	379	519	1082	492	394	481	720	231
GlossaryDisambiguation		7	3	5	7	3	2	2	4	2
GlossaryList 🗑			2				1	1
htmlattrdef		82	44	38	82	67	58	61	83	19
htmlattrxref		405	112	144	340	121	91	146	218	41
HTMLElement	2	1518	417	596	1193	449	374	533	828	178
HTMLRef		160	59	161	160	119	85	100	147	26
HTMLRefTable		1	1	1	1	1	1	1	1	1
HTMLSidebar		62	26	57	50	33	16	18	34	5
HTMLVersionInline 🗑			19	1			22	14
httpheader		423	83	110	309	123	159	122	222	48
HTTPMethod		71	24	25	67	47	49	41	56	15
HTTPSidebar		314	86	165	250	122	183	123	224	55
HTTPStatus		74	30	25	76	47	62	51	66	22
IncludeSubnav 🗑			35				18	45
InheritanceDiagram		731	23	64	165	39	16	30	92	24
Interwiki 🗑			135		323		104	136
js_property_attributes		45	34	50	50	28	39	40	47	5
JSFiddleEmbed		30	21	12	14	12	1	26	30	4
jsOverrides 🗑							5	5
JSRef		680	312	665	647	344	355	409	535	107
JsSidebar		269	162	262	260	194	158	220	253	101
jsxref	1	2588	559	948	1478	674	573	783	1178	231
LandingPageListSubpages		14	12	11	16	9	10	11	9	11
languages 🗑			111				19	17
LearnSidebar		313	160	187	207	141	136	186	218	119
Link 🗑			1			1	1	2
ListGroups		2	2	2	2	2	2	2	2	1
ListSubpages	1	6	5	3	7	4	6	5	5	5
ListSubpagesForSidebar	4	142		7	34	2			6	1
LiveSampleLink		12	3	2	12	3	2	3	8	2
LiveSampleURL	2
MathMLElement		48	6	23	38	3	5	9	20	2
MathMLRef		43	1	9	25		1	9	15
MDNSidebar		85	23	30	75	15	24	20	30	13
mediawiki.external 🗑			19	5	5	3	2	4
minversionGeneric	1
Next		7	6	6	6	7	5	6	7	5
NextMenu		26	19	22	20	17	17	18	23	13
nextPage 🗑							1
no_tag_omission		71	25	35	72	70	37	43	67	13
Non-standard_Header		487	109	169	167	16	34	52	107	4
Non-standard_Inline		214	67	117	101	62	66	105	123	45
Non-standardGeneric	1			1
NonStandardBadge	8	1	1	2	1				2
Note 🗑			8				8	11
NoteEnd 🗑	1		3				2	1
NoteStart 🗑	1		3				2	1
Obsolete_Header 🗑			11				17	14
Obsolete_Inline 🗑			41		1		47	65
ObsoleteBadge	7
ObsoleteGeneric 🗑	2		6				6	3
optional_inline		853	77	404	371	130	67	95	271	48
optionalGeneric	1
page			1				53	78	67	28
Previous		5	4	5	3	4	2	4	5
PreviousMenu		25	11	15	19	10	9	16	20	6
PreviousMenuNext	2	184	92	129	125	85	76	112	136	70
PreviousNext	2	88	53	75	82	58	28	72	83	44
previousPage 🗑							1
propertiesbox	1
property_prefix 🗑					1	1		2
QuickLinksWithSubpages		123	21	38	83	32	7	33	31	6
ReadOnlyInline		735	94	203	272	104	89	142	279	66
RFC		112	62	51	144	38	131	92	27	6
SectionOnPage		130	11	9	22	15	10	20	36	7
secureContext_header		512	20	57	150	12	13	16	39	2
secureContext_inline		23	3	7	16	6	3	8	4	1
secureContextGeneric	2
SeeCompatTable		1150	161	320	426	81	78	159	440	42
ServiceWorkerSidebar		2	1	2	2	2	2	2	2
source			2		6		2	2
spec 🗑			12	1	11	1	7	11
spec2 🗑		1	971	1117	1363	56	938	535
Specifications		7691	105	1873	2362	1451	63	1090	3329	494
SpecName 🗑		1	1044	1132	1809	64	1025	544
SubpagesWithSummaries	1	9	6	7	4	2	2	3	4	1
SVGAttr		325	18	84	93	18	56	50	137	3
SVGElement	2	480	19	101	114	22	59	65	180	5
svginfo		80	9	62	36	4	45	18	79
SVGRef		326	11	95	66	8	54	32	109
TemplateLink 🗑			3				3	2
TenthCampaignQuote		1	1		1	1	1	1	1	1
unimplemented_inline		5	8	10	5	6	5	15	18
unimplementedGeneric	1
warning 🗑		1	3	1	1		2	1
web.link 🗑			47				1	3
WebAssemblySidebar		77	3	6	11	11	4	8	10	1
WebExtAllCompatTables		1	1	1	1		1		1
WebExtAllExamples		1	1	1	1	1	1	1	1
WebExtAPIRef		488	7	414	148	22	22	24	67	14
WebExtAPISidebar	1
WebExtExamples		547	4	505	118	20	17	11	59	8
WebGLSidebar		18	8	12	18	11	4	10	18	6
WebkitBug	1	3	3	1	4	1	2	1	2	1
WebRTCSidebar		22	2		2	7	2	11	8
xref_csscomputed	1	1	6		1				1	2
xref_cssinherited	1	1	6		1				1	2
xref_cssinitial	1	1	6		1				1	2
xref_cssvisual		1	16		1		1		1	2
XsltRef		102	52	95	85	75	1	1	5
XULAttr 🗑			1	4	3
XULElem 🗑			1	3	4	1

How did you test this change?

Ran yarn -s macro-usage-report.

Sep 07 '22 16:09 caugner

@yin1999 @teoli2003 @SphinxKnight This might be interesting for you.

Sep 07 '22 17:09 caugner

@yin1999 Would you like to open two PRs for removing CompatAndroid and CSSTutorialTOC?

React to this comment with 👀 when you (or anybody else) starts working on it.

Sep 07 '22 18:09 caugner

Strangely, I found a call to xref_cssvisual in https://github.com/mdn/content/blob/main/files/en-us/web/css/-webkit-mask-box-image/index.md while the script returns none.

(We should hardcode the table there, so that we can deprecate 4 more macros)

Sep 08 '22 08:09 teoli2003

Strangely, I found a call to xref_cssvisual in https://github.com/mdn/content/blob/main/files/en-us/web/css/-webkit-mask-box-image/index.md while the script returns none.

Good catch, ~~it looks like the parentheses in Xref_cssvisual() are considered part of the word, so it doesn't match xref_cssvisual\b~~ TIL that String.split() with a limit just drops the remainder (in contrast to php's explode()).

Sep 08 '22 08:09 caugner

Strangely, I found a call to xref_cssvisual in https://github.com/mdn/content/blob/main/files/en-us/web/css/-webkit-mask-box-image/index.md while the script returns none.

@teoli2003 I have now fixed that bug, thanks for reporting. 👍

Sep 08 '22 09:09 caugner

This pull request has merge conflicts that must be resolved before it can be merged.

Sep 09 '22 09:09 github-actions[bot]

Maybe we should ignore _redirects.txt, for there is a redirect like this (maybe a mistake):

https://github.com/mdn/translated-content/blob/8ed6dea31e84cf67e2f413a6f47ba751350abc13/files/fr/_redirects.txt#L1847

/fr/docs/HTML/Element/Output_{{HTMLVersionInline(5)}}_{{fx_minversion_inline(4)}}	/fr/docs/Web/HTML/Element/Output

Or we can remove this redirect? /cc @SphinxKnight

Oct 09 '22 08:10 yin1999

Maybe we should ignore _redirects.txt, for there is a redirect like this (maybe a mistake):

https://github.com/mdn/translated-content/blob/8ed6dea31e84cf67e2f413a6f47ba751350abc13/files/fr/_redirects.txt#L1847
/fr/docs/HTML/Element/Output_{{HTMLVersionInline(5)}}_{{fx_minversion_inline(4)}}	/fr/docs/Web/HTML/Element/Output
Or we can remove this redirect? /cc @SphinxKnight

let me get to it, this one scratches like an itch :)

Oct 09 '22 10:10 SphinxKnight

Redirects removed w/ https://github.com/mdn/translated-content/pull/9343

Oct 19 '22 12:10 SphinxKnight

Fails if CONTENT_TRANSLATED_ROOT is not set.

Good catch, fixed in bf171dfd9.

Dec 08 '22 10:12 caugner

Had a side thought about this. What about instead of adding this to the CLI itself, just adding CI cron job + manual trigger to the repo for this, and have it create/update an issue?

Dec 08 '22 17:12 nschonni

Had a side thought about this. What about instead of adding this to the CLI itself, just adding CI cron job + manual trigger to the repo for this, and have it create/update an issue?

Adding a workflow for this is an idea worth pursuing, but rather out of the scope of this PR. You would probably still need the CLI command itself, but given that it can produce JSON with --format=json, it might even be possible to automatically open or update a PR removing all unused macros. Also, I wouldn't be sure that rg is available on GitHub runners (maybe it is).

Dec 08 '22 18:12 caugner

Different approach, since it's used to comment on each PR, but something you can copy a little for the commenting with action/github-script if you decide to try that later https://github.com/w3c/aria-practices/blob/main/.github/workflows/coverage-report.yml

Dec 08 '22 18:12 nschonni

@nschonni Can you elaborate your proposal? I thought you were suggesting to automatically open an issue as soon as a macro is no longer used. :)

Dec 08 '22 21:12 caugner