Automation analysis

[simcha-vos-from-tu-delft]

I'm with the Software Engineering Research Group at the TU Delft, and I am investigating automations. I have looked at your repository and which GitHub workflows and Maven plugins you are using. I'll give you a quick summary of what I found, and the automation tasks I recommend you to focus on next (:

Level of maturity	Basic	Intermediate	Advanced
Collaboration	‎ ✅ Completed this level!
	✔️ Prepare or create documentation artifacts	✔️ Generate documentation from source code ❌ Commit validation ❌ Bot commits	✔️ Publish documentation ❌ Issues or PRs management
Code quality	‎ ⚠️ Still working on this level!
	❌ Run tests ❌ Static code style analysis	❌ Test coverage and validity ❌ Generate test reports ❌ Automatic code formatting ❌ Static code quality analysis	❌ Verify packaging correctness ❌ Vulnerability scans ❌ Sign artifacts ❌ License checks
Development	‎ ⚠️ Still working on this level!
	✔️ Build environment configuration ❌ Package management	✔️ Build files configuration	✔️ Optimization
Artifacts	‎ ✅ Completed this level!
	✔️ Code compilation ✔️ Dependency management of artifact	✔️ Build tasks, resources and configuration ✔️ Packaging ✔️ Release tagging ✔️ Publish artifacts to a registry ❌ Generate source and metadata artifacts ❌ Source code version control	❌ Generate release notes ❌ Containerization ❌ Push container to remote

And now? Next steps!

It is not always clear which automation tasks should be prioritized. It is however important to balance your automation efforts, as a uniform level of maturity is most productive. I'm here to help! Below is a list of tasks that you can work on to help level up your maturity across the automation domains:

• Implement Package management; implemented by 36% of GitHub repositories
• Implement Run tests; implemented by 23% of GitHub repositories

What do you think?

Do you think my analysis is correct, do these recommendations help you? Or have I missed something?

For my master's thesis I'm doing research into the use of automations in GitHub repositories. With my findings, I want to see if I can help out developers with what automations they could focus on. Do you want to help me out? Leaving a response is much appreciated!

[bact]

@simcha-vos-from-tu-delft Thank you.

From a quick look, maybe the "Run tests" detection has missed the test by either Sonar or mvn verify.

https://github.com/spdx/Spdx-Java-Library/blob/0906908bf452d3c106a8dfc953cff3d92a314d44/.github/workflows/build.yml#L42-L47

[simcha-vos-from-tu-delft]

@simcha-vos-from-tu-delft Thank you.

From a quick look, maybe the "Run tests" detection has missed the test by either Sonar or mvn verify.

Spdx-Java-Library/.github/workflows/build.yml

Lines 42 to 47 in 0906908

   run: | 
     if [[ $SONAR_TOKEN != "" ]]; then 
        mvn clean org.jacoco:jacoco-maven-plugin:prepare-agent install sonar:sonar 
     else 
        mvn --batch-mode --update-snapshots verify 
     fi

Thanks for your reply! Indeed, I found it to be hard to always consistently detect everyone's tests. Thanks for showing me this part (:

[simcha-vos-from-tu-delft]

@simcha-vos-from-tu-delft Thank you.

From a quick look, maybe the "Run tests" detection has missed the test by either Sonar or mvn verify.

Spdx-Java-Library/.github/workflows/build.yml

Lines 42 to 47 in 0906908 run: | if [[ $SONAR_TOKEN != "" ]]; then mvn clean org.jacoco:jacoco-maven-plugin:prepare-agent install sonar:sonar else mvn --batch-mode --update-snapshots verify fi

Hello, I have one final question to help me better understand your interest in tools like this! Given that this tool would be detecting all your automations properly, would you be interested in using it, as a way to easily report on your current state of automations and gain recommendations for future automation steps? For example, if its available as a badge in your readme? Thank you!

[bact]

Thanks for your reply! Indeed, I found it to be hard to always consistently detect everyone's tests. Thanks for showing me this part (:

The Sonar part maybe too customised but I think mvn verify is pretty a standard one.

For a question about badge, I think in general it will be like all badges that the maintainers will use it in two different ways (separately and together): 1) to remind themselves on a quality goal to achieve and 2) to present the current quality of their project to others.

Since your evaluation is about the controls/processes, I think we can say that the badge is built upon the belief that a good product comes from a good process. So (2) will happen more for the products that target users that believe in this assumption. (2) can drive (1). But (1) can occurs independently as well.

[goneall]

@simcha-vos-from-tu-delft Thanks for adding the report - I'm always looking for feedback to improve the quality of the libraries I maintain.

In addition to the static code analysis and testing mentioned above, I believe the source and metadata artifact files are being generated as part of the release artifacts - let me know if I missed something.

I also have a question on the "package management" under development - what exactly is the purpose / expectation of automating this step? It is in the release automation, is it expected to be on a per-PR merge? If so, can you explain why?

We are also generating SPDX SBOM files on release which I would encourage you to add to your automation. SBOMs provide a detailed representation of the code dependencies and license information and is a requirement for several industries.

[bact]

@simcha-vos-from-tu-delft for license check, apart from information in an SBOM file, a file header is another venue. SPDX provides a standardised way to embed copyright and license information into text file:

// SPDX-FileContributor: Streeling University
// SPDX-FileCopyrightText: 2019 Jane Doe <[email protected]>
// SPDX-FileType: SOURCE
// SPDX-License-Identifier: GPL-2.0-or-later

See: https://spdx.github.io/spdx-spec/v2.3/file-tags/