Use area labeler bot

[danmoseley]

The labeler bot currently in the benevolent care of the dotnet/runtime team is used in many repos to apply "area-" style labels to issues and PR's based on an ML model over text, author, and files touched. With enough to learn on, it's highly accurate.

I think there's more than enough "area-" labeling prior art in this repo now to train and enable the bot. Even better if before the first training someone does some bulk labeling based on keywords - I've done this in other repos eg SDK but it's not needed unless you see the bot flailing.

Any interest?

[danmoseley]

After it's enabled, the cost to this repo is zero. Although, if the bot does make a bad guess, it's best to correct it, so that it can improve when retrained some time in the future. Similarly its nice, but not necessary, to correctly label any the bot skips because it can't guess. We do these things in dotnet/runtime.

[benvillalobos]

Any interest?

Bug Triage: Is it easy to enable this feature? It's not critical for us but if it can be turned on easy we'd like to see how it works out for us.

[danmoseley]

@BenVillalobos should be. We have a new hire starting next week who will be taking over ownership of the bot, and I'll connect with him. He can reach out ..

[benvillalobos]

Sounds good.

As an example, I would expect something like https://github.com/dotnet/msbuild/issues/7368 to be marked as Task: GenerateResource

[danmoseley]

Cc @dakersnar for later

[maryamariyan]

As an example, I would expect something like https://github.com/dotnet/msbuild/issues/7368 to be marked as Task: GenerateResource

@BenVillalobos two questions:

• Do all the labels that need predicting start with a certain prefixes? e.g. in your example Task: ? I also see some labels prefixed with Area: and Area-.
• If an issue or PR is labeled Task: GenerateResource can it also be labeled Area: Debuggability for example? Asking because the issue labeler applies only one label at a time.

[danmoseley]

Cc @Eilon . We maybe should encourage some consistency across repos (eg prefix with 'area-')

[maryamariyan]

FYI, Originally the labeler bot filtered through labels prefixed with area- it has been since extended further to accept area-, area/ and area:.

Refer to https://github.com/Eilon/issue-labeler/blob/main/src/CreateMikLabelModel/DL/LabelHelper.cs

[rainersigwald]

I'm personally not tied to the label formats we currently have, but I think it makes sense to preserve a couple of types of label:

• reason/tenet type labels, like performance or debuggability, describing how an issue affects users, and
• feature/code location type labels, like engine or Task: GenerateResource, describing what part of the codebase is relevant.

For the latter, MSBuild has many tasks, and it's helpful to distinguish between them. It looks like runtime does this with additional - delimited words in the tag, so our Task: GenerateResource could correspond to something like area-tasks-generateresource.

[benvillalobos]

Do all the labels that need predicting start with a certain prefixes? e.g. in your example Task: ? I also see some labels prefixed with Area: and Area-.

imo we should move to dash-separated labels like area-foo. I prefer lowercase labels, but it looks like label queries on github are case-insensitive.

If an issue or PR is labeled Task: GenerateResource can it also be labeled Area: Debuggability for example? Asking because the issue labeler applies only one label at a time.

Thinking in terms of someone trying to find bugs to work on: someone may want to improve the GenerateResource task and specifically not care about debuggability, so I'd say these should be two separate labels.

[Eilon]

To summarize and solidify some thoughts about area labels:

• What are they for?
  • Area labels help teams working in repos to identify what part of the team or what feature area an issue or PR belongs to.
  • Area labels can work in tandem with other automation that can alert the owners of an area that an issue has come up and to be triaged
• Application of labels:
  • The goal is that every issue and PR in the repo has an area label. If there's no reasonable existing area label, perhaps you need a new area label.
  • There should be as little overlap between area labels as possible so that issues/PRs fit "nicely" into a single area
  • Corollary: Issues/PRs should thus typically have only one area label applied. If an issue has more than one area label, consider whether it should be split into multiple issues so that it can be triaged by each applicable area owner. It's certainly allowed to have more than one area label, but it should be avoided so that it's clear what part of the team is accountable.
• Label names in the repo:
  • Use any of these patterns area: XYZ, area-XYZ, or area/XYZ, because that's what our tools already look for. Introducing new patterns has associated cost and risk.
  • Casing is unimportant because GitHub labels are case-insensitive everywhere.
  • Recommended: Include a description of the label to aid humans to understand what the label represents. When you select a label in the GitHub UI, the text search includes the label name and description:
    
  • Color: Whatever works for you. Some teams use one color for all area labels, some use whatever color they want for each label

To be clear, this isn't gospel, so anyone can do it however they want, but I think this pattern matches what most teams do with area labels, and fits best with a lot of existing tools and processes.

[Eilon]

Hi everyone, great news! I have run some training on the MSBuild repo and you can try to "manual" mode using Hubbup (a site I built) for applying labels based on the best predictions: https://hubbup.io/miklabel/MSBuild (sign in with GitHub). Here's what it looks like:

The list shows all issues and PRs that have no "area" labels. It shows the best predicted label using a web service that was trained on the existing issues/PRs in the repo. You can also just pick any other label if the best predictions aren't right.

Next steps:

1. If you want to keep using the manual labeling, go for it! (AspNetCore and MAUI use that)
2. If you want to using the WebHook fully automated labeling, let me know, and I'll try to figure out how to enable that (we have no instructions, so it might take a while 😁)
3. Once you've used it for a while and applied many more labels, we should re-train the label model. This takes like 5 minutes of effort, so let me know when you want to do it, and we can do it

And please let me know any feedback you have on any part of this system, process, behavior, interface, etc.!

[danmoseley]

Curious how this is going ..

[rainersigwald]

The first round of auto-suggestions weren't very good, so I've been trying to label stuff in the hopes that another round of training will help.

[benvillalobos]

I'm starting to dip my toes into correcting some of the labels ✌️

[Eilon]

Once you've done some labeling, please let me know and I will re-run the training so that you can hopefully get better predictions as a result. It only takes like 30 minutes to do.

[benvillalobos]

@Eilon Just did another pass!

[Eilon]

The training data has been updated and is live. The previous update (the initial list) was from May 26 and added up to 860KB of data, and now it's about 1010KB of data, so hopefully that helps the predictions.

BTW there are very few PRs with area labels so those predictions are probably lower quality than issue predictions. There are 620 issues with area labels, and only 122 PRs. There's no magic number of labeled issues/PRs that makes things magically great, but more is always better.

[danmoseley]

When I've done mass labeling in other repos to jump start the labeler I was able to label many 100's in a relatively short time: I searched for a distinctive keyword that would almost always imply a certain label, then control clicked so all the pages of results were in different tabs, then on each page checked off everything that seemed like the label applied and chose the label, then while it was plowing through setting it on each of those I did the same to the next page, etc. then repeat for another distinctive keyword.

[danmoseley]

@benvillalobos @Eilon where are we at with this one -- I don't thnk I see it labeling things? Or did it stop? If the repo owners would still like to try this, should we do one more training and switch it on?

[Eilon]

I never heard from any MSBuild people that they wanted it enabled. I'm still happy to help with that if anyone in this repo wants to use either the fully-automated system, or the human-accepts-prediction system.