orleans
orleans copied to clipboard
dotnet
Fix CI actions and optimize them a bit
@ReubenBond Maybe this is a start, or a sketch, how to start with GitHub builds. Do tell if you have ideas, and of course this can be abandoned as a learning tool, modified now, merged and worked on more later (of course, that's why we're here :)) and the usual caveats.
I fixed the branches in the GH Actions here, added macos-latest as seemed to be in Azure DevOps, added also .NET 5, tests and such. The build times are fairly long. If tests were excluded, turnaround time were faster. One option could be also to run only a substet of tests (but which one?). Considering which .NET frameworks to support, how to run tests etc. could have an impact to build times. But this "refactoring tests" thing is, of course, a larger story.
I'll check about filters and running steps in parallel in multiple steps, maybe excluding those that will fail anyway.
@veikkoeeva what happens if we add the test categories into the mix? Functional, SlowBVT, and BVT are the categories we use. GitHub Actions seems to struggle with these - loading the logs for one of these failed runs is taking many minutes already and is still not finished. Azure DevOps may be more suitable option in the end
@ReubenBond We have to see if something could and should be filtered out. Are Azure DevOps tests possible to run in forked repos, by the way? I think I'll try those next, also add security tests as a parallel job to see how it goes.
I am not sure how the DevOps tests run, but are they on a private/on-premises runner?
In any case, I think it would be nice to introduce Docker for the databases and run tests to them. It looks like requiring some changes to how the tests run. I am thinking just plain starting a storage engine on command line and then running tests with a connection string to that particular engine.
Added quickly some filtering and security scanning to see the effect.
I think building is needed only once for the tests and scanning to use the artefacts after they've been built. So consequently that is one job for building (build) and then two new jobs need to be added, e.g. test, security-scan. So using https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idneeds for this, code scanning probably can happen only on one platform.
@ReubenBond If it is OK to keep it open as I doodle. Now indeed that you poked and I remembered there was this going on.
Let I put here I welcome discussion or anyone else working on this.
Here is a short summary
I think the core goal is to build and run tests on .NET 6 and consider how to make them quick and modular as far as build is concerned.
Other goals that this evolved into are:
- Take a look how to run the build agent locally (in Docker) for quicker build pipeline editing. This may be needed first.
- Consider how to to fix flaky tests.
- Security testing with CodeQL, Stryker and other that come into mind carrying towards Nuget and deployments.
- I'll link also https://github.com/actions/virtual-environments/discussions/4421, partially because of previous item.
- Take a look how to run ADO.NET tests in the pipeline. At least for one database.
- Take a look how to run Docker images in tests, for e.g. more exotic integration testing.
I originally lost a bit of momentum as I wanted to look a way to run the build pipeline locally. I ran out of disk space (fixed that with a bigger SSD) and eventually time. Then this evolved also to more secure coding and CodeQL did not have .NET 6 support (it has now). Then I went to another repo to add Stryker (it had bugs in .NET 6, fixed already, yay!)). So this evolved to something like Hal fixing a light bulb.
I recognize not all of these things may not be what is desirable right now, but I'd like to map out how it would look. Thus far it feels the builds are very slow and it might be useful to think how to enable builds to run locally as a first thing.