Add Integration Testing

[colincmac]

As we start to implement new features and have more developers, unit tests will be necessary to avoid breaking functionality. Also, this tool is provided as-is, without Microsoft support (outside of the team here). It will encourage developer adoption.

[aristosvo]

Hi @colincmac @waelkdouh! As a user and (potential) contributor it would make life a lot easier if we had something like this. I'm not sure whether it should be a Unit Test, feels more like this project needs integration testing.

As I'm not sure how testing is done currently behind the scenes I feel a bit reluctant to contribute Integration Testing for the whole project, could you give a bit of insight in what is going on behind the scenes with regard to testing?

My suggestion would be to make the tests used public, as it would make contributing easier. If there is nothing yet, I could make a start with:

• Providing Bicep/scripts to set up a starting situation (APIM Dev + Prod and Azure DevOps/GH)
• Create a .NET test project which does things like:
  • Create repos in AzDO and GH
  • Create APIs in Dev Instance
  • Create AzDO pipelines/GH workflows
  • Change APIs in a Dev Instance
  • Check if everything works out
  • Cleans it up

This is a huge undertaking and would be only worth it if it has your support from the start. What is your view on the matter?

[waelkdouh]

I will let @guythetechie speak to this as I know he started working on integration testing starting yesterday.

[guythetechie]

Thanks for your feedback and your willingness to contribute, @aristosvo. I agree that this project needs integration tests, as we currently don't have any. They should have been there from the start, but better late than never...

As @waelkdouh mentioned, I started playing around with this yesterday. Here's what I had in mind:

Extractor tests

Goal: Validate that extracted data matches instance data.

Arrange

1. Create APIM instance with test data (APIs, policies, etc).
2. Compile extractor binaries.

Act

1. Run extractor against APIM instance.

Assert

Make sure extracted resources match APIM instance state. Something like:

1. For each APIM resource we support:
   • Get resource state in APIM instance
   • Get extracted resource (JSON, policy XML, etc)
   • Make sure they match

Publisher tests

Goal: Validate that artifacts are published successfully.

Arrange

1. Create a clean APIM instance.
2. Create an artifacts folder with test data, like this one.
3. Compile extractor and publisher binaries.

Act

1. Run publisher with the test data.

Assert

1. Run extractor on published instance.
2. For each APIM resource we support, make sure extracted artifact matches test data artifact. Using extractor here gives extra sanity checks that extractor is working.

For both of these tests, even just having the Arrange/Act sections work successfully would be a big improvement. Would love to hear some feedback.

Note: I felt .NET was too "low-level" for my publisher tests, so I went with PowerShell and Pester. Here's a proof of concept:

BeforeDiscovery {
    $parameters = @{        
        testArtifactsPath = "C:\Users\user\source\repos\github\Azure\apiops\tools\tests\artifacts"
        extractorSourcePath = "C:\Users\user\source\repos\github\Azure\apiops\tools\code\extractor\extractor.csproj"
        publisherSourcePath = "C:\Users\user\source\repos\github\Azure\apiops\tools\code\publisher\publisher.csproj"
    }
}

Describe "Publisher tests" -ForEach $parameters {
    BeforeAll {
        Set-StrictMode -Version Latest
        $ErrorActionPreference = "Stop"
        $VerbosePreference = "Continue"
        $InformationPreference = "Continue"
    
        Write-Information "Creating resource group..."
        $resourceGroupParameters = @{
            Name     = "apiops-test-rg"
            Location = "eastus"
            Force    = $true
        }
        $resourceGroup = New-AzResourceGroup @resourceGroupParameters

        Write-Information "Creating APIM instance..."
        $apimInstanceParameters = @{
            "ResourceGroupName" = $resourceGroup.ResourceGroupName
            "Name"              = "apiopststr-apim"
            "Location"          = $resourceGroup.Location
            "Organization"      = "apiops"
            "AdminEmail"        = "[email protected]"
            "Sku"               = "Consumption"
        }
        $apimInstance = New-AzApiManagement @apimInstanceParameters

        Write-Information "Compiling extractor..."
        $extractorOutputFolderPath = Join-Path $TestDrive "extractor"
        & dotnet publish $extractorSourcePath --self-contained --runtime "win-x64" -p:PublishSingleFile=true --output "$extractorOutputFolderPath"
        $extractorFilePath = Join-Path $extractorOutputFolderPath "extractor.exe"

        Write-Information "Compiling publisher..."
        $publisherOutputFolderPath = Join-Path $TestDrive "publisher"
        & dotnet publish $publisherSourcePath --self-contained --runtime "win-x64" -p:PublishSingleFile=true --output "$publisherOutputFolderPath"
        $publisherFilePath = Join-Path $publisherOutputFolderPath "publisher.exe"    
    
        Write-Information "Running publisher..."
        $env:AZURE_BEARER_TOKEN = (Get-AzAccessToken).Token
        $env:AZURE_SUBSCRIPTION_ID = (Get-AzContext).Subscription.Id
        $env:AZURE_RESOURCE_GROUP_NAME = $apimInstance.ResourceGroupName
        $env:API_MANAGEMENT_SERVICE_NAME = $apimInstance.Name            
        $env:API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH = $testArtifactsPath        
        $output = & $publisherFilePath
        if ($LASTEXITCODE -ne 0) {
            throw "Publisher failed. Output is $($output | Out-String)"
        }
            
        Write-Information "Running extractor..."
        $extractorArtifactsPath = Join-Path $TestDrive "extractor artifacts"
        $env:API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH = $extractorArtifactsPath        
        $output = & $extractorFilePath
        if ($LASTEXITCODE -ne 0) {
            throw "Extractor failed. Output is $($output | Out-String)"
        }
    }

    Context "Policy fragments" -ForEach $parameters {
        BeforeDiscovery {
            $policyFragmentsFolder = Join-Path $testArtifactsPath "policy fragments"
            $policyFragments = $policyFragmentsFolder | Get-ChildItem -Directory
            $parameters = $policyFragments | ForEach-Object {
                @{
                    name          = $_.Name
                    artifactsPath = $_.FullName
                }
            }
        }

        It "Publishes <_.name> XML successfully" -ForEach $parameters {
            $testXmlPath = Join-Path $artifactsPath "policy.xml"
            $testXmlContent = Get-Content -Path $testXmlPath -Raw

            $extractorXmlPath = Join-Path $extractorArtifactsPath "policy.xml"
            $extractorXmlContent = Get-Content -Path $extractorXmlPath -Raw

            $extractorXmlContent | Should -Be $testXmlContent
        }
    }
}

[aristosvo]

Nice, simple and AAA, like it.

Would you keep the pipelines out of scope? Seems a reasonable choice for now.

[guythetechie]

Yep, I think we can leave the pipelines out of scope for now.