[tests] Workload tests

[radical]

These would be tests that exercise the workload, like:

Template tests

• [ ] test for aspire-apphost, and aspire-servicedefaults - take the ApiService, and Web project from the aspire-starter template and then add apphost, and servicedefaults from templates
• [ ] with no workload installed, a project with $(IsAspireHost)=true would fail to build due to missing aspire workload
• [x] basic templates can build and run
• [x] Starter template can run (WIP)
• [x] Starter template's included test project works (WIP)
• [x] create+build+run starter template, and run some basic checks including checking the dashboard (WIP)

Dashboard

• [x] verify all the resources show up (WIP)

Others

• [x] This is being handled indirectly with playground, and EndToEnd tests. ~~Test per published nuget, to use the nuget and perform some basic validation~~
• [ ] manifest publishing works with dotnet run
• [x] default https works on linux
• [x] ASPIRE_ALLOW_UNSECURED_TRANSPORT

Future:

[radical]

cc @joperezr

[eerhardt]

If I was dreaming, we would also have tests that publish an app to Azure. But that is going to take a lot of infrastructure.

with no workload installed, a project with $(IsAspireHost)=true would fail to build due to missing aspire workload

I'd put this tests pretty low on the priority list.

[mitchdenny]

I'd love to have tests that work on Azure as well. In the Azure SDK repo they have the ability to do something like this: /azp run net - servicex - tests

This is a dedicated pipeline that only people with appropriate permissions can trigger which results in provisioning resources in Azure. Because the identity that provisions the resources is an Owner of the sub it uses a dedicated sub for this purpose, and the identity is restricted in what it can access.

You also need to have a bot that goes through and deletes resources.

@radical if you are looking at this I recommend going and having a chat to @weshaggard about what their current state of the art is on the Azure SDK side.

[radical]

This sounds interesting. thoughts, @eerhardt ?

[mitchdenny]

I think the test matrix is going to be fairly interesting. For example, do you want to test Aspire across multiple clouds and regions. API versions can sometimes take time to rollout so if we adopt a new version of an ARM API via CDK and publish our packages it's possible that we could block someone from upgrading their app in a region which tends to receive API updates late - similar for different clouds. China and USGov (not to mention others) can lag behind significantly.

We probably will also want to look at real world upgrade scenarios. This probably means that in our repo we'll need small free standing applications that have been built using various versions of the Aspire stack and then have scripts that sequentially deploy the updates to make sure we don't break upgrade scenarios.

These would be a bit different to the playgrounds, and unit tests aren't good candidates for this because the "earlier versions" won't be using the latest versions of packages but would be tied to the latest servicing release for that generation.

[eerhardt]

What is the medium-to-long-term vision for where the Azure Hosting code lives? If we intend to move it to https://github.com/Azure/azure-sdk-for-net/, it may make sense to wait until then to hook it up to automated tests.

On the Components side, my goal is to eventually move them to the Azure repo, so they can ship and service along with the Azure SDK packages.

[radical]

Combined the remaining tasks in #2689 and https://github.com/dotnet/aspire/issues/6076 into https://github.com/dotnet/aspire/issues/9576 . Closing.