.NET 8/9: External product dependencies governance

[premun]

This epic is defined based on an investigation/proposal/design done as part of the Unified Build epic, the business continuity effort and the licensing validation plan. The decisions driving the need for this epic are captured in the design document:

📄 Design / Strategy for managing external Source-Build dependencies

Business objective

This epic focuses on ensuring that any source code that is part of the Source-Build is on par with the standards we hold for the rest of the code. Namely, this means that every external dependency complies with the following requirements:

1. Serviceability
   We must be able to apply any patch onto any dependency, behind closed doors, if need be (to allow security patches). Furthermore, it is expected it might not be possible to upstream all the patches as they might be customizations related to integration of the component into Source-Build and would not be suitable for upstreaming. Additionally, we should have a clear audit trail of changes made.
2. Business continuity
   Thinking long-term, we must ensure the ability to build non-current versions of the product for servicing reasons. Even in a case when a third-party dependency is no longer available.
3. Source-Buildability
   It must be possible to build each external dependency that is part of the Source-Build from source. a. The source files of the dependency must be stripped from disallowed artifacts (binary files). b. We validate the component builds from source early in the pipeline (ideally at the point of ingestion).
4. Secure supply chain
   We should not assume that owners of the dependencies adhere to the same level of due diligence as us. Ultimately, this means we scan the sources of the dependencies for security threats at the point of ingestion.
5. Licensing compliance
   We must ensure that third party components are compliant from a licensing perspective.
6. Auditability
   For each source-build dependency introduced into the product, we need to keep records of its origin so that origin of the source code can be exactly identified. Additionally, we need to keep track of all changes applied to the original source code by us.
7. Frequent synchronization with upstream
   The goal is to stay as close as possible with the dependency’s upstream to ensure the code flows both ways: a. Downstream to consume feature updates, security patches and bug fixes. b. Upstream to honor the OSS playbook to contribute back to the original project.

Technical objectives

The design document elaborates on this more, but when we translate these requirements into actual work needed, we end up with the following list:

• [ ] Identify external dependencies Source-Build dependencies There are three ways how we ingest external dependencies (external submodules, forked submodules, vendoring). We need to identify all dependencies that are part of the Source-Build, categorize them and determine owners.
• [ ] Prepare tooling for external dependency management We will need tooling that ensures satisfaction of the requirements (CG scan, license validation, ...) that we can provide to teams so that they can onboard their dependencies. We need automation to validate compliancy breaches where possible.
• [ ] Prepare guidance/docs for consuming external dependencies Teams onboarding new dependencies or teams that want to change how they consume already existing dependencies should have guidance at disposal that will help them weigh benefits/requirements/drawbacks of different ingestion types so that they can make an educated decision.
• [ ] Prepare/enforce policy for external dependency management Each dependency, based on how it's ingested in the product, requires a little bit different approach to become compliant. We need to have documentation ready so that we can ask teams to comply with the requirements.