project-system icon indicating copy to clipboard operation
project-system copied to clipboard
verified publisher icon dotnet

Migrate integration tests to internal repository and build/run process

Open MiYanni opened this issue 3 years ago • 2 comments

Currently, integration tests do not function. The legacy MyGet package source is no longer referenced by the repo, and the tooling required for APEX (very old version) was being pulled from that source. All the APEX tooling (new versions) are published internally. A majority of integration tests exist within the VS repo itself, but as we all know, using that repo is not a fun endeavor. Additionally, integration tests run on an unreleased version of VS (in-progress development versions), so there is no avenue where it makes sense to have integration tests to be ran/exposed publicly.

My proposal is as such:

  • Create an internal repo that only contains our integration tests
    • This keeps us out of the VS repo (🥳)
    • Integration tests need the internal APEX packages to be built/developed
  • Create a pipeline for building/publishing the integration tests
    • Integration tests should also be accessible to OptProf, which requires them to be published to a certain location
  • Create a pipeline for running the integration tests
    • Grabs the latest published version of the tests (as opposed to building them itself)
    • Grabs the target project-system PR/version (can pull from the build produced by another pipeline)
    • Runs the tests
    • Determine how results are provided (maybe a bot that produces comment on the GitHub PR?)
  • Determine when/how integration tests are run
    • Per PR? Schedule? During Integration? Some of these depend on how long it takes to run the tests.

MiYanni avatar Jul 27 '22 23:07 MiYanni

Looks like there is a way to configure Apex tests publicly, but they run through DartLab. The configuration script is produced as part of a build artifact, so it doesn't actually exist in the repo. https://github.com/NuGet/NuGet.Client/blob/dev/eng/pipelines/templates/Apex_Tests_On_Windows.yml

This is an option I wasn't aware of, and wouldn't have existed prior to this push to use DartLab. It is using the same template that we use for OptProf, except it clearly configured differently. This seems quite involved, and without looking deeper, I don't know where the tests themselves actually reside.

MiYanni avatar Sep 21 '22 23:09 MiYanni

I've added the details and progress on this work here: https://dev.azure.com/devdiv/DevDiv/_wiki/wikis/DevDiv.wiki/34643/Integration-Testing-(in-progress)

MiYanni avatar Jun 07 '23 00:06 MiYanni