How to merge multiple open api specification files in publisher pipeline?

[Shafeeqts89]

Release version

Azure APIM v6.0.0

Question Details

Is it possible to merge multiple open API specification files in the publisher pipeline?

Expected behavior

We have a specification file for a set of standard operations and another specification file for custom operations. we need to merge these files and deploy it to APIM

Actual behavior

As expected Only one open API specification file is deployed

Reproduction Steps

.

[github-actions[bot]]

  Thank you for opening this issue! Please be patient while we will look into it and get back to you as this is an open source project. In the meantime make sure you take a look at the [closed issues](https://github.com/Azure/apiops/issues?q=is%3Aissue+is%3Aclosed) in case your question has already been answered. Don't forget to provide any additional information if needed (e.g. scrubbed logs, detailed feature requests,etc.).
  Whenever it's feasible, please don't hesitate to send a Pull Request (PR) our way. We'd greatly appreciate it, and we'll gladly assess and incorporate your changes.

[waelkdouh]

We only provide the pipelines as starter pipelines. It's up to you to modify them as you see fit. As long as the publisher finds that file to publish it then you should be fine. What happens prior to that can be altered as you see fit. Just make sure the format of your directory is intact as the publisher expects a certain structure which is what the extractor is providing.

[Shafeeqts89]

I have tried open api merge cli to merge two open API specification files and it seems merging is working as expected. However, pipeline is throwing the below error. it would be great if you could provide some suggestions.

crit: Publisher[0] System.IO.DirectoryNotFoundException: Could not find a part of the path '/home/vsts/work/1/s/artifacts'. at System.IO.Enumeration.FileSystemEnumerator1.CreateDirectoryHandle(String path, Boolean ignoreNotFound) at System.IO.Enumeration.FileSystemEnumerator1.Init() at System.IO.Enumeration.FileSystemEnumerable1..ctor(String directory, FindTransform transform, EnumerationOptions options, Boolean isNormalized) at System.IO.Enumeration.FileSystemEnumerableFactory.FileInfos(String directory, String expression, EnumerationOptions options, Boolean isNormalized) at System.IO.DirectoryInfo.InternalEnumerateInfos(String path, String searchPattern, SearchTarget searchTarget, EnumerationOptions options) at System.IO.DirectoryInfo.EnumerateFiles(String searchPattern, EnumerationOptions enumerationOptions) at common.ArtifactDirectoryExtensions.EnumerateFilesRecursively(IArtifactDirectory directory) at publisher.Publisher.RunWithoutCommitId(CancellationToken cancellationToken) at publisher.Publisher.Run(CancellationToken cancellationToken) at publisher.Publisher.ExecuteAsync(CancellationToken cancellationToken) info: Microsoft.Hosting.Lifetime[0] Application is shutting down... Unhandled exception. System.IO.DirectoryNotFoundException: Could not find a part of the path '/home/vsts/work/1/s/artifacts'. at System.IO.Enumeration.FileSystemEnumerator1.CreateDirectoryHandle(String path, Boolean ignoreNotFound) at System.IO.Enumeration.FileSystemEnumerator1.Init() at System.IO.Enumeration.FileSystemEnumerable1..ctor(String directory, FindTransform transform, EnumerationOptions options, Boolean isNormalized) at System.IO.Enumeration.FileSystemEnumerableFactory.FileInfos(String directory, String expression, EnumerationOptions options, Boolean isNormalized) at System.IO.DirectoryInfo.InternalEnumerateInfos(String path, String searchPattern, SearchTarget searchTarget, EnumerationOptions options) at System.IO.DirectoryInfo.EnumerateFiles(String searchPattern, EnumerationOptions enumerationOptions) at common.ArtifactDirectoryExtensions.EnumerateFilesRecursively(IArtifactDirectory directory) at publisher.Publisher.RunWithoutCommitId(CancellationToken cancellationToken) at publisher.Publisher.Run(CancellationToken cancellationToken) at publisher.Publisher.ExecuteAsync(CancellationToken cancellationToken) at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token) at publisher.Program.Main(String[] arguments) at publisher.Program.<Main>(String[] arguments) Exception: /home/vsts/work/_temp/6aba5254-5723-4f25-a2e9-6a39f5fabd07.ps1:9 Line | 9 | if ($LASTEXITCODE -ne 0) { throw "Running publisher failed."} | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | Running publisher failed.

And below is my publisher yaml file:

trigger:
  branches:
    include:
      - main  # Trigger only for changes in the main branch
  paths:
    exclude:
    - tools/*

parameters:
  - name: API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH
    type: string
    displayName: Folder where the artifacts reside
    default: "artifacts"
  - name: COMMIT_ID
    type: string
    displayName: Choose "publish-all-artifacts-in-repo" only when you want to force republishing all artifacts (e.g. after build failure). Otherwise stick with the default behavior of "publish-artifacts-in-last-commit"
    default: publish-artifacts-in-last-commit
    values:
      - publish-artifacts-in-last-commit
      - publish-all-artifacts-in-repo

variables:
  - group: apim-automation
  - name: System.Debug
    value: true

stages:
  - stage: push_changes_to_Dev_APIM
    displayName: Push changes to Dev APIM
    jobs:
      - job: merge_and_push_changes
        displayName: Merge and Push Changes to Dev APIM
        pool:
          vmImage: ubuntu-latest
        steps:
          - checkout: self  # Ensure the repository is checked out

          - script: |
              echo Checking the contents of "$(Build.SourcesDirectory)/BO APIOps Test/openapi"
              ls -la "$(Build.SourcesDirectory)/BO APIOps Test/openapi"
            displayName: 'Check OpenAPI Directory Contents'

          
          - script: |
              npm install -g openapi-merge-cli
            displayName: 'Install OpenAPI Merge CLI'
          - script: |
              echo 'Creating the merge configuration file...'
              echo '{
                "inputs": [
                  {
                    "inputFile": "BO APIOps Test/openapi/contact.json"
                  },
                  {
                    "inputFile": "BO APIOps Test/openapi/lead.json"
                  }
                ],
                "output": "BO APIOps Test/artifacts/apis/Contacts/specification.json"
              }' > $(Build.SourcesDirectory)/openapi-merge-config.json
            displayName: 'Generate OpenAPI Merge Configuration'
            workingDirectory: $(Build.SourcesDirectory)
          - script: |
              openapi-merge-cli -c openapi-merge-config.json
            displayName: 'Merge API Specifications'
            workingDirectory: $(Build.SourcesDirectory)

          - script: |
              echo "Checking current working directory..."
              pwd
              echo "Checking if the directory exists and listing contents..."
              ls -lah "BO APIOps Test/artifacts"
            displayName: 'Further Debug Directory Access'

          - script: |
              cat "$(Build.SourcesDirectory)/BO APIOps Test/artifacts/apis/Contacts/specification.json"
            displayName: 'Display Merged File Content'
          - task: PublishBuildArtifacts@1
            displayName: 'Publish Merged Specification as Artifact'
            inputs:
              PathtoPublish: "$(Build.SourcesDirectory)/BO APIOps Test/artifacts/apis/Contacts/specification.json"
              ArtifactName: 'MergedOpenAPISpec'
              publishLocation: 'Container'
          
                                           
          - template: run-publisher-with-env01.yaml
            parameters:
              API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH: ${{ parameters.API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH }}
              RESOURCE_GROUP_NAME: $(RESOURCE_GROUP_NAME_DEV)
              API_MANAGEMENT_SERVICE_NAME: $(APIM_NAME_DEV)
              ENVIRONMENT: "Dev"
              COMMIT_ID: ${{ parameters.COMMIT_ID }}

[Shafeeqts89]

It worked when I changed the API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH parameter from $(Build.SourcesDirectory)/${{ parameters.API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH }} to "BO APIOps Test/artifacts"

Since I'm a beginner in yaml pipelines, I couldn't find any other issues with the yaml.

[guythetechie]

The publisher needs to know where your artifacts are located. It gets that value from the configuration value API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH. By default, our starter pipelines set it to $(Build.SourcesDirectory)/artifacts here, here, and here.

You can change this as per your comment above, or by changing this to BO ApiOps Test/artifacts.

[Shafeeqts89]

@guythetechie When I merge and deploy openapi files using the above pipeline the following error is thrown.

info: Publisher[0] Putting policy policy for operation post-contacts in api Contacts... crit: Publisher[0] System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/APIOps-PoC/providers/Microsoft.ApiManagement/service/BO-APIOps-Poc/apis/Contacts/operations/get-contacts-contactid/policies/policy?api-version=2022-04-01-preview&format=rawxml failed with status code 400. Content is '{"error":{"code":"ValidationError","message":"Entity with specified identifier not found","details":null}}'. at common.HttpPipelineExtensions.Validate(Response response, Uri requestUri) at common.HttpPipelineExtensions.PutResource(HttpPipeline pipeline, Uri uri, JsonObject resource, CancellationToken cancellationToken) at publisher.Program.<>c__DisplayClass13_0.<<GetPutRestResource>b__0>d.MoveNext() --- End of stack trace from previous location --- at publisher.ApiOperationPolicy.PutPolicy(ApiOperationPolicyName policyName, ApiOperationName operationName, ApiName apiName, JsonObject json, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken) at publisher.ApiOperationPolicy.<>c__DisplayClass9_0.<<ProcessArtifactsToPut>b__0>d.MoveNext() --- End of stack trace from previous location --- at System.Threading.Tasks.Parallel.<>c__541.<<ForEachAsync>b__54_0>d.MoveNext() --- End of stack trace from previous location --- at common.IAsyncEnumerableExtensions.ForEachParallel[T](IAsyncEnumerable1 enumerable, Func2 action, CancellationToken cancellationToken) at publisher.ApiOperationPolicy.ProcessArtifactsToPut(IReadOnlyCollection1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken) at publisher.Service.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, GetRestResource getRestResource, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken) at publisher.Publisher.RunWithoutCommitId(CancellationToken cancellationToken) at publisher.Publisher.Run(CancellationToken cancellationToken) at publisher.Publisher.ExecuteAsync(CancellationToken cancellationToken) info: Microsoft.Hosting.Lifetime[0] Application is shutting down...

The strange thing is that the same merged file from the above pipeline is successfully deployed through the starter pipeline. any thoughts?

[guythetechie]

Can you try again with our latest v6 release? There was a related issue that was addressed in the release.

[daviian]

I've come across the same problem again, using the latest v6 release (v6.0.1.1)

[shawnmurtagh]

@Shafeeqts89 Have you tried using your merge tool manually and trying to paste your OAS into the azure portal? That might give you a validation error or something better than the pipelines. If it works, you could also extract it and do a file compare to see what APIM did to refactor your OAS with some of it's opinions.

It's been a while - you may have already solved this but I was curious!