dotnet
Add API scan to the list of SDL tools run
- [ ] This issue is blocking
- [ ] This issue is causing unreasonable pain
As part of compliance efforts many teams around DevDiv have to run API Scan on their binaries, and it would be great if Arcade could provide this value where possible, similar to how CredScan and PoliCheck are provided now.
An API Scan build task is available so hopefully its possible to do: https://www.1eswiki.com/wiki/APIScan_Build_Task
Is this something that belongs in the release epic @adiaaida and @epananth
This seems more like an FR task to me, as it doesn't just apply to releases (ie, it sounds like something that would happen in official builds, not in post-build scenarios).
@davidwengier can you give more details about how/where this is being used today and what it's meant to be run on?
how/where this is being used today
DevDiv has regular compliance checkpoints every year, and at other times like major releases (eg, Dev16), and as part of that teams have various work items created that instruct them to run tools like API Scan, CredScan, PoliCheck etc. For example the following tasks were created from this years assessments:
https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1316735 https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1316432 https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1316485
For repos that use Arcade the CredScan and PoliCheck requirements are already covered which saves teams a heap of time in dealing with these checks (as well as the obvious benefit of ensuring the the software is always correct throughout development, rather than having to scramble and fix things at the last minute).
Add API Scan to the tools that Arcade runs would be a big help, and is possibly the only remaining piece of these compliance checkpoints that is automatable.
what it's meant to be run on?
API Scan just runs against the binaries produced by a build. For example I found this PR that enables it for the StreamJsonRpc repo: https://github.com/microsoft/vs-streamjsonrpc/pull/659/files
Thank you so much for the added information! That link to the repo that already has enabled it is really helpful.
@markwilkie I think that we should definitely consider adding this to the validate/staging pipelines, however, that will not cover everyone, as not all of our product repos have decided to onboard to the validation pipeline. For getting it added to arcade/post-build and enabled for repos outside of the nightly validation pipeline, that feels more like an FR task to me.
My thought is that our guidance is to use the validation/staging pipelines to participate in automation etc.. If however product teams want to do something in a different way, they are of course free to do so. (no FR needed)
Adding this to the release with confidence epic, with the goal to add it to the Validate and Staging pipelines
I think too that Guardian is (or is planning) on adding this. Not super sure the details though or even if what I'm saying is correct.
Any news on this? We're just wrapping up another compliance checkpoint for DevDiv and this would have saved a few teams a bunch of effort :)
Just wanted to chime in on David's question above. We are starting another compliance checkpoint for DevDiv and gearing up to run API scan manually for now. My team owns some internal AD repos that use arcade and that would really benefit from having this tooling. Thanks!
I just started a task to upgrade Guardian in the Arcade repos (#8075).
I don't have experience with API Scan, but that's a fixable problem, and this is a good time to look at adding the support. I'll take a look and see what it will take.
I see that Guardian offers this as part of their DevOps Extension. I think that will carry over to the CLI (which is what Arcade uses). If yes, great. If not, we'll have to explore a different approach. I'll investigate.
@mmitche and I met with the PMs, who requested that they have a way to automate APIScan (currently run manually right before releases). So, I'm bringing this in to the Operations Epic for consideration.
With the split of dnceng and dnceng-public, we can now use the Guardian Azure DevOps extension, which should make running compliance-related tasks much easier than the hand-crafted CLI and YAML template juggle we do now.
Originally, we discussed the idea of attaching this to the nightly release pipeline as a straightforward to cover the entire product. But, this makes reporting tricky because the execution will look like a single "repository" and thus a single "owner"; it won't be able to, say, assign issues to appropriate teams. There has also been a desire by repos that don't participate in the nightly pipeline but do use Arcade to also have access to APIScan.
So, I wonder if it would be straightforward to drop the APIScan Guardian Task (in the sense of an Azure DevOps Pipeline Task) into the post-build template in Arcade. At a glance, adding the YAML to call the task is all it should take (with some plumbing to expose whatever configuration makes sense).
The Ask is to have something, not necessarily final, in place by .NET 8 Preview 1.
@garath I think it was Preview 1? Anyways, the main issue is that most of the core repos don't use the post-build template. They use the nightly validation pipeline for SDL. How is reporting done for that right now?
I think it was Preview 1?
Oops. You are right. Fixed.
Anyways, the main issue is that most of the core repos don't use the post-build template. They use the nightly validation pipeline for SDL. How is reporting done for that right now?
Hm, I'm not sure. I'll check when I can dig into this issue more. I'm sure we can make it work.
@garath looks like it's been a little while since this issue was touched. Do you recall what the action items on it are?
I believe this is now a "Product Construction" issue, and I also believe Tomas's team has already made some progress on it?
@tkapin @premun should we move this to Arcade Services? Or is there someone actively working on it that it should be assigned to and left in the "Ops" backlog?
@andriipatsula and @MilenaHristova work on https://github.com/dotnet/arcade-services/issues/2647, which was picked by us as it was related to the release infrastructure (staging & validation pipelines), even though it's at the end also enabled on the product repos.
Should this API scan be set on the repo level? If so, it's rather a shared tooling (arcade) rather than product construction, which is related to how the repos are assembled together.
I believe ensuring APIScan happens in the release pipeline is sufficient for the overarching requirement of "shipping .NET".
Individual repos may use the APIScan build task, which has recently been enabled in the dnceng/internal Azure DevOps project (along with many other Guardian tools). Dnceng has a service principal available for use if needed (APIScan requires an onboarded principal as part of the scan).
@garath - "I believe ensuring APIScan happens in the release pipeline is sufficient for the overarching requirement of "shipping .NET"." - can you be more specific about this? Are you referring to the recent BinSkim efforts done per https://github.com/dotnet/arcade-services/issues/2647 or it's something else yet?
I'm re-opening this to make sure we don't lose track of this until the question above is answered.
@tkapin, yep, I refer to the efforts from arcade-services#2657. My original involvement with this was to automate the manual run done by PMs just before release. This seems to meet that requirement.
Issue has gone idle, so closing. Feel free to reopen if there are further questions.