[Improve Best Practice Guide]: Security scanning with SCRUB, et. al.

[riverma]

Checked for duplicates

Yes - I've already checked

Best Practice Guide

Continuous Testing

Best Practice Guide Sections

• [X] Use Cases
• [X] Starter Kits

Describe the improvement

We'd like to integrate NASA's SCRUB into a continuous build system to make it easy for projects to automate security scanning, per NASA recommendations.

[riverma]

@jpl-jengelke - as @awdtinio mentioned, please chat with @jeffreypon as he's experimented with SCRUB.

Also - I think the security scanning with SCRUB ticket should really be delivered as a Jenkins plugin for people to quickly / easily adapt to their existing projects. Having a plug-and-play plugin would likely reduce the amount of time developers would have to spend getting SCRUB scanning initialized for their own CI. Take a look at this guide: https://www.jenkins.io/doc/developer/tutorial/create/

Thoughts?

[jpl-jengelke]

There's a Maven plugin that will integrate with the CodeSonar system, but of course Maven is mostly Java-specific. Developing a Jenkins plugin is a big task, so I don't think by any measure that it's doable with the resources at our disposal. However, we can offer a demonstration that is configurable and plug-and-play enough to cover all bases. That is what I am working on completing.

[riverma]

+1'd by @ramesh-maddegoda, @jeffreypon, @pymonger

[jordanpadams]

@riverma FYI, @lylebarner helped us get this integrated into our CodeQL execution using GitHub Actions. here is an example: https://github.com/NASA-PDS/validate/blob/main/.github/workflows/codeql-analysis.yml#L65

[riverma]

Hi @jordanpadams - thanks for pointing us to @lylebarner and the GH action you all have built for validate. This is pretty cool.

@jpl-jengelke - let's definitely work together and discuss with @jordanpadams and @lylebarner on synergizing their existing code with the goals of this ticket as we work on this during Q2.

[jeffreypon]

AMMOS currently setup to use Codesonar and sonarqube. Scrub outputs a .csv report in a specific format that the cybersecurity team can ingest into their VATT tool to perform adjudication.

@lylebarner If new tools are added to scrub (i.e. CodeQL), does the scrub tool output the .csv results in the specific format that can be imported into the cybersecurity's VATT tool without any software changes? Or does scrub have to be modified for each tool to get it in the specific format?

[jeffreypon]

@jpl-jengelke Make sure to grab the latest version of scrub. There was a major update this past week that changed some of the .csv output formats. The cybersecurity team is requesting all scans be performed using the latest version of scrub now. https://github.com/nasa/scrub

[jpl-jengelke]

Make sure to grab the latest version of scrub. Thank you!

[riverma]

There's another ticket we should probably merge into this one: https://github.com/NASA-AMMOS/slim/issues/51

[riverma]

Some ideas discussed with @jpl-jengelke for this ticket:

• Add CI integration to regularly and automatically invoke SCRUB to Java Starter Kit
• Add CI integration to regularly and automatically invoke SCRUB to Python Starter Kit

[jpl-jengelke]

Proposed implementation

1. CI Server (Jenkins) Implementation (Java example)
   1. Execute inline as a post-build pipeline step
   2. Run against SonarQube
   3. Publish report in Jenkins workspace
   4. Publish artifact in Jenkins workspace
2. GitHub Action Implementation (Python Example)
   1. Isolate execution in standalone GH Action (separate container)
   2. Run against CodeQL and PyLint
   3. Publish formatted report alongside Action Summary

flowchart LR
    subgraph CI["CI Server (Exa.: Java)"]
        C1[Compile]-->C2[Test]
        C2-->C3[Package]
        C3-->C4["Publish\n(Artifactory)"]
        C2-.-> |"Success, Monthly"| C5[SCRUB]
    end
    subgraph GA["GitHub Action (Exa.: Python)"]
        A1[Compile]-->A2[Test]
        A2-->A3[Package]
        A3-->A4[SCRUB]
        A4-->A5["Release\n(PyPi)"]
    end

[riverma]

Thanks for the write-up diagram @jpl-jengelke! Some questions:

• What's preventing, if anything, SCRUB to execute at the test stage instead of post build? I think that would be a great value-add for teams to be more preventative with security scanning as early as possible than to possibly have public builds out there with security issues.
• From @ramesh-maddegoda's question: can we justify why SonarQube and CodeQL / PyLint are the right tools to run via SCRUB? Why not other tools that are institutionally or open-source-environment supported by SCRUB's wrapping framework?

[riverma]

@lylebarner - had a few questions for you:

• Related to the question above, do you have recommendations for how to filter for a smaller sample of SCRUB scanning results such that SCRUB can run tools like SonarQube / CodeQL quickly enough on individual Git commits?
• Are there projects you can point us to that have successfully integrated SCRUB with Jenkins, e.g. for Java minimally?

[lylebarner]

• Related to the question above, do you have recommendations for how to filter for a smaller sample of SCRUB scanning results such that SCRUB can run tools like SonarQube / CodeQL quickly enough on individual Git commits?

There is an institutional Secure Coding Guideline that can be used for this. Generally this is considered a good starting place for identifying high priority security issues. This guide can be used to tailor down to a smaller set of CWEs. I can provide a link via email.

• Are there projects you can point us to that have successfully integrated SCRUB with Jenkins, e.g. for Java minimally?

Florin Tudor has set up several other repos with SCRUB-based scanning on Jenkins. He should be able to provide you with more information or at least point you in the right direction.

[jpl-jengelke]

• What's preventing, if anything, SCRUB to execute at the test stage instead of post build?

Timing. We can't add anything to extend the build times for VICAR products since the timeline is so unpredictable in my experience. Potentially this could eat 30+ minutes, and that would not meet users' level of service expectations. In current planning we wanted this to occur once a month, maybe quarterly, at most, just to avoid long build times.

Now for a POC in SLIM we could do that, but we wouldn't want someone implementing it in a VICAR product right now.

[jpl-jengelke]

Thanks @lylebarner I'll check in with Florin about it.

[lylebarner]

@jpl-jengelke would it be possible to perform SCRUB analysis in parallel with test activities? From a technical point of view, SCRUB only needs the source code itself (and compiled outputs such as .class/.jar files) to perform analysis. Not sure of the system resources that are required to run the test suite though.

[jpl-jengelke]

@jpl-jengelke would it be possible to perform SCRUB analysis in parallel with test activities? From a technical point of view, SCRUB only needs the source code itself (and compiled outputs such as .class/.jar files) to perform analysis. Not sure of the system resources that are required to run the test suite though.

Yes, it is possible, though some accommodations must be made. One point of the test phase in the Maven lifecycle is to provide an exit point if tests fail, before packaging. So the SCRUB operations would be run against .class files but not packaged .jar files, which are actually the final deliverable.

The tasks would be run in containers to provide a cleanroom environment that isolates them from the CI server setup.

Another option is to run in a parallel job but that would likely require a second compilation.

Thanks for the write-up diagram @jpl-jengelke! Some questions:

• What's preventing, if anything, SCRUB to execute at the test stage instead of post build? I think that would be a great value-add for teams to be more preventative with security scanning as early as possible than to possibly have public builds out there with security issues.

Costs versus value. During a release cycle, code should be turned around fast. If we execute SCRUB at the test stage then potentially tens of minutes are added to what must be a fast-turnaround build-compile-release cycle. The artifact is not generated until the test cycle completes (not sure if I'd add a pass/fail test). There are options but for a process that generates a report intermittently, the accommodation is to run it at the end of packaging and delivery.

I think we can more readily apply this to the testing stage for a Python deliverable that is completely generated and released using GitHub Actions.

(Note: (rough definitions) packaging == convert .class files into JARs; delivery == push compiled binaries (JARs) to an artifact repository; release == pushing the product out to servers or for public distribution.)

• From @ramesh-maddegoda's question: can we justify why SonarQube and CodeQL / PyLint are the right tools to run via SCRUB? Why not other tools that are institutionally or open-source-environment supported by SCRUB's wrapping framework?

Part of the charge of this project is to make tools that support Lab processes and requirements. Because CodeQL is the successor to Semmle, and the Lab security reports are predicated on Semmle, we do need to support it. Otherwise, these are soft recommendations based on conventions (as understood) that we can re-evaluate with trade studies. For instance, SonarQube is a very Java-centric tool that is highly adopted and more critically implements an interface that combines a number of underlying code analytic tools, such as PMD, Jacoco, etc.

[jpl-jengelke]

Note, preliminarily modified design flow (above) to accommodate comments after our meeting. I'll be sending out a follow up invitation today. Thanks everyone!

[riverma]

Thanks for the feedback and responses @jpl-jengelke!

I think @lylebarner's thoughts on a potentially running scans in parallel to builds is really interesting. Perhaps it's better to run scans nightly / on a schedule as opposed to tying to build triggers? The length of the code scan issue you mentioned is the same problem that comes up often for running system regression tests - which can take hours. Thoughts?

[sharimayer]

Thinking about our Vicar builds in particular and knowing that only the source code is needed to run Scrub, it might make sense to run Scrub before the nightly builds (like 7 p.m.) as the nightly builds kick off at 9 p.m.

[riverma]

Thanks for that @sharimayer. Quick question: if no source code has changed, do the nightly builds still get triggered? If not - what mechanism is used to know nothing has changed?

[sharimayer]

Yes, the nightly builds get triggered even if no source code has changed.

-- Shari C. Mayer IDS CM

From: Rishi Verma @.> Date: Wednesday, March 8, 2023 at 11:55 PM To: NASA-AMMOS/slim @.> Cc: Mayer, Shari C (US 398G) @.>, Mention @.> Subject: [EXTERNAL] Re: [NASA-AMMOS/slim] [Improve Best Practice Guide]: Security scanning with SCRUB, et. al. (Issue #25)

Thanks for that @sharimayerhttps://urldefense.us/v3/__https:/github.com/sharimayer__;!!PvBDto6Hs4WbVuu7!N-3XbT8BcKx28EF1DesE1MXtuG8N4FzNTqcclJgoXCLfcL56KgWBP8OFt5jKZQMCtu7k2CK3XsBl05UR7rwU0BLWxno99_U$. Quick question: if no source code has changed, do the nightly builds still get triggered? If not - what mechanism is used to know nothing has changed?

— Reply to this email directly, view it on GitHubhttps://urldefense.us/v3/__https:/github.com/NASA-AMMOS/slim/issues/25*issuecomment-1461506381__;Iw!!PvBDto6Hs4WbVuu7!N-3XbT8BcKx28EF1DesE1MXtuG8N4FzNTqcclJgoXCLfcL56KgWBP8OFt5jKZQMCtu7k2CK3XsBl05UR7rwU0BLW710VFJU$, or unsubscribehttps://urldefense.us/v3/__https:/github.com/notifications/unsubscribe-auth/AKQIMYHRTEHXG6GMZVTLC6DW3GEF5ANCNFSM5UTJYRDA__;!!PvBDto6Hs4WbVuu7!N-3XbT8BcKx28EF1DesE1MXtuG8N4FzNTqcclJgoXCLfcL56KgWBP8OFt5jKZQMCtu7k2CK3XsBl05UR7rwU0BLWw50jmn0$. You are receiving this because you were mentioned.Message ID: @.***>

[jpl-jengelke]

From @riverma: Quick question: if no source code has changed, do the nightly builds still get triggered? If not - what mechanism is used to know nothing has changed?

From @sharimayer: Yes, the nightly builds get triggered even if no source code has changed.

@sharimayer is correct.

However, in the Java VICAR ecosystem the SCRUB scans will take place at compile/build/publish time on the CI server -- which is much earlier when changes in code are pushed to Git. It's a lean system that is set right now to compile on demand when it sees changes. As a result Java scans will take place before the VICAR nightly builds. (Nightly builds pull previously compiled JARs from the artifact repository, e.g. Artifactory.)

For VICAR C builds, a compile/re-compile also takes place every night, whether something changes or not, and that is the point at which SCRUB scans are expected to take place in the C world.

We haven't touched much on the C world yet and this task is focused mainly on automating the Java and Python side by augmenting the CI server processes, e.g. Jenkins, or the GitHub Actions side, e.g. tooling in the source code repository server.

[jpl-jengelke]

From @riverma: I think @lylebarner's thoughts on a potentially running scans in parallel to builds is really interesting. Perhaps it's better to run scans nightly / on a schedule as opposed to tying to build triggers? The length of the code scan issue you mentioned is the same problem that comes up often for running system regression tests - which can take hours. Thoughts?

We'll surely discuss this at today's meeting but that is a legitimate solution to the conundrum of running the job after every build. It does, however, decouple it from code commits and compiles so feedback is not immediate. However, it's an open question whether the SCRUB scan process requires immediate intervention or feedback. I would argue that most similar processes, such as Dependabot security scanning, are run on timers. In many respects that's a policy question.

[jpl-jengelke]

Thanks Everyone for attending today's meeting. It was truly informative and inspirational to hear everyone's take on this task, and I believe we came to a consensus on how to proceed.

Takeaways:

• SCRUB analyses will be in separate actions/files
• Single configurations may be available to be applied generically to projects, making it easy to clone.
• Developers can run SCRUB with the correct configuration at any time.
• SCRUB not only does security scans, it also does static code analysis by default, which is valuable for developers.
• Caution is noted as sometimes tooling breaks and we would not want to disrupt a release due to a broken analysis application.

After this meeting, I will re-draw the above workflow and proceed with developing the discussed solutions.

Proposed solution:

• Python Starter Kit
  • separate action
  • action runs on both dev commit and tag release
  • keep CSV file close to action
  • Create magic link that can be referenced from repo entrypoint README to get to latest SCRUB results
  • possibly explore options for GUI reporting
  • will not break compile/release/push
• Java Starter Kit
  • separate Jenkinsfile
  • Jenkinsfile runs on a timer at a regular interval or on demand
  • Jenkinsfile re-uses configured auth settings for build/compile/release Jenkinsfile
  • On Lab, jobs will reference the SonarQube server which provides a GUI report

Please feel free to comment, ask questions or add additional information. Great meeting today!

[riverma]

@jpl-jengelke: recommendation next steps for this ticket (two separate PR's):

1. A PR write-up containing directions for installing a GitHub Action to execute SCRUB automatically
2. A PR write-up to execute SCRUB within a local Jenkins environment

[riverma]

See @lylebarner's ticket over at SCRUB to help with (ii): https://github.com/nasa/scrub/issues/99

[riverma]

@jpl-jengelke: recommendation next steps for this ticket (two separate PR's):

1. A PR write-up containing directions for installing a GitHub Action to execute SCRUB automatically
2. A PR write-up to execute SCRUB within a local Jenkins environment

@jpl-jengelke - I think we have (1) from @lylebarner write-up - we can point to this. Do you have a working, generic, solution to document for (2)? Having the two in place - we should be able to close this ticket soon.

[jpl-jengelke]

Yes I do. I will publish it shortly within the Java Starter Kit. That will close a number of tickets simultaneously.