[BUG] ApiPolicy Put Error: Entity with specified identifier not found

[daviian]

Release version

5.1.4

Describe the bug

When publishing an API spec and an API Policy at the same time the APIM returns "Entity with specified identifier not found" as a result of the API Policy HTTP request. After some experiments it looks like it only happens for larger (> ~1.5k LOC) specifications. I checked the specification, and tested it by first publish the API Spec and afterwards in another commit the API Policy - in such a case it works without an issue. So to me it looks like as if it is a timing issue, but I also noticed that a long polling in the code should take care of this.

fail: Microsoft.Extensions.Hosting.Internal.Host[9]
      BackgroundService failed
      System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/<redacted>/resourceGroups/<redacted>/providers/Microsoft.ApiManagement/service/<redacted>/apis/my-test-api/policies/policy?api-version=2023-05-01-preview&format=rawxml failed with status code 400. Content is '{"error":{"code":"ValidationError","message":"Entity with specified identifier not found","details":null}}'.
         at common.HttpPipelineExtensions.Validate(Response response, Uri requestUri)
         at common.HttpPipelineExtensions.PutResource(HttpPipeline pipeline, Uri uri, JsonObject resource, CancellationToken cancellationToken)
         at publisher.Program.<>c__DisplayClass13_0.<<GetPutRestResource>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at publisher.ApiPolicy.PutPolicy(ApiPolicyName policyName, ApiName apiName, JsonObject json, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken)
         at publisher.ApiPolicy.<>c__DisplayClass7_0.<<ProcessArtifactsToPut>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at System.Threading.Tasks.Parallel.<>c__54`1.<<ForEachAsync>b__54_0>d.MoveNext()
      --- End of stack trace from previous location ---
         at common.IAsyncEnumerableExtensions.ForEachParallel[T](IAsyncEnumerable`1 enumerable, Func`2 action, CancellationToken cancellationToken)
         at publisher.ApiPolicy.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken)
         at publisher.Service.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, GetRestResource getRestResource, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken)
         at publisher.Publisher.RunWithoutCommitId(CancellationToken cancellationToken)
         at publisher.Publisher.Run(CancellationToken cancellationToken)
         at publisher.Publisher.ExecuteAsync(CancellationToken cancellationToken)
         at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)

Expected behavior

API spec and API Policy are published fine within a single commit.

Actual behavior

API Policy is not published and publisher stops (does not update the rest of the resources)

Reproduction Steps

I was able to reproduce it by using larger specifications, something above 1.5k LOC

[github-actions[bot]]

  Thank you for opening this issue! Please be patient while we will look into it and get back to you as this is an open source project. In the meantime make sure you take a look at the [closed issues](https://github.com/Azure/apiops/issues?q=is%3Aissue+is%3Aclosed) in case your question has already been answered. Don't forget to provide any additional information if needed (e.g. scrubbed logs, detailed feature requests,etc.).
  Whenever it's feasible, please don't hesitate to send a Pull Request (PR) our way. We'd greatly appreciate it, and we'll gladly assess and incorporate your changes.

[guythetechie]

@daviian - thanks for raising this. We caught this issue as well in our integration tests for the next release. APIM sometimes fails to assign a policy if done right after the resource is created.

Fix should be in the next release.

[daviian]

@guythetechie - do you have an estimate for when this release will be completed?

[daviian]

@guythetechie any update on this topic?

[waelkdouh]

Please subscribe to the release notifications on the repo to be notified when a new release is pushed. No exact timelines available at this point.

[waelkdouh]

we just tested under v6.0.0-rc1 and the issue has been resolved. Please test and let us know.

[daviian]

Unfortunately this issue is still present. I got the same error today with version v6.0.1.1.

crit: publisher[0]
  Application failed.
  System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/<redacted>/resourceGroups/<redacted>/providers/Microsoft.ApiManagement/service/<redacted>/apis/<redacted>/operations/<redacted>/policies/policy?api-version=2023-05-01-preview failed with status code 400. Content is '{"error":{"code":"ValidationError","message":"Entity with specified identifier not found","details":null}}'.
    at common.HttpPipelineExtensions.<>c__DisplayClass11_0.<PutContent>b__0(Response response) in /home/runner/work/apiops/apiops/tools/code/common/Http.cs:line 173
    at LanguageExt.Either.Left`2.Match[B](Func`2 Left, Func`2 Right)
    at LanguageExt.Either`2.IfLeft(Func`2 leftMap)
    at common.HttpPipelineExtensions.PutContent(HttpPipeline pipeline, Uri uri, BinaryData content, CancellationToken cancellationToken) in /home/runner/work/apiops/apiops/tools/code/common/Http.cs:line 173
    at common.ApiOperationPolicyModule.PutDto(ApiOperationPolicyUri uri, ApiOperationPolicyDto dto, HttpPipeline pipeline, CancellationToken cancellationToken) in /home/runner/work/apiops/apiops/tools/code/common/ApiOperationPolicy.cs:line 132
    at publisher.ApiOperationPolicyModule.<>c__DisplayClass11_0.<<GetPutApiOperationPolicyInApim>b__0>d.MoveNext() in /home/runner/work/apiops/apiops/tools/code/publisher/ApiOperationPolicy.cs:line 181
  --- End of stack trace from previous location ---
    at publisher.ApiOperationPolicyModule.<>c__DisplayClass7_1.<<GetPutApiOperationPolicy>b__1>d.MoveNext() in /home/runner/work/apiops/apiops/tools/code/publisher/ApiOperationPolicy.cs:line 123
  --- End of stack trace from previous location ---
    at common.OptionExtensions.IterTask[T](Option`1 option, Func`3 action, CancellationToken cancellationToken) in /home/runner/work/apiops/apiops/tools/code/common/Option.cs:line 22
    at common.OptionExtensions.IterTask[T](Option`1 option, Func`2 action) in /home/runner/work/apiops/apiops/tools/code/common/Option.cs:line 16
    at publisher.ApiOperationPolicyModule.<>c__DisplayClass7_0.<<GetPutApiOperationPolicy>b__0>d.MoveNext() in /home/runner/work/apiops/apiops/tools/code/publisher/ApiOperationPolicy.cs:line 123
  --- End of stack trace from previous location ---
    at publisher.ApiOperationPolicyModule.<>c__DisplayClass1_1.<<GetPutApiOperationPolicies>b__2>d.MoveNext() in /home/runner/work/apiops/apiops/tools/code/publisher/ApiOperationPolicy.cs:line 58
  --- End of stack trace from previous location ---
    at common.IEnumerableExtensions.<>c__DisplayClass2_0`1.<<IterParallel>b__0>d.MoveNext() in /home/runner/work/apiops/apiops/tools/code/common/Enumerable.cs:line 40
  --- End of stack trace from previous location ---
    at System.Threading.Tasks.Parallel.<>c__53`1.<<ForEachAsync>b__53_0>d.MoveNext()
  --- End of stack trace from previous location ---
    at common.IEnumerableExtensions.IterParallel[T](IEnumerable`1 enumerable, Func`3 action, Int32 maxDegreeOfParallelism, CancellationToken cancellationToken) in /home/runner/work/apiops/apiops/tools/code/common/Enumerable.cs:line 61
    at common.IEnumerableExtensions.IterParallel[T](IEnumerable`1 enumerable, Func`2 action, CancellationToken cancellationToken) in /home/runner/work/apiops/apiops/tools/code/common/Enumerable.cs:line 40
    at publisher.ApiOperationPolicyModule.<>c__DisplayClass1_0.<<GetPutApiOperationPolicies>b__0>d.MoveNext() in /home/runner/work/apiops/apiops/tools/code/publisher/ApiOperationPolicy.cs:line 54
  --- End of stack trace from previous location ---
    at publisher.AppModule.<>c__DisplayClass1_0.<<GetRunApplication>b__0>d.MoveNext() in /home/runner/work/apiops/apiops/tools/code/publisher/App.cs:line 186
  --- End of stack trace from previous location ---
    at common.HostingModule.RunApplication(IHost host) in /home/runner/work/apiops/apiops/tools/code/common/Hosting.cs:line 89

[daviian]

I've looked into it and the culprit is an invalid openapi spec and an HTTP response status 'OK' instead of 'Accepted' when uploading the API spec.

Due to this JsonOpenApiSpecificationFile.TryParse() returns Option<OpenApiSpecVersion>.None resulting in the specification to not being uploaded at all. This exception however is silently handled and the user is not aware of the actual root cause at all. Since the spec is not uploaded, no operations exists which obviously fails the policy upload request.

Furthermore the second Put request in common\Api.PutNonSoapApi():623 does not respond with an HTTP status 'Accepted' but with 'OK', so the WaitForLongRunningOperation is not doing anything. Looks like this was a change in API Version 2023-09-01-preview. Using API Version 2022-08-01 does the trick and everything is fine.

This could also be the issue of #553

EDIT: Proposal: instead of returning None the exception should be propagated so that the user is informed about the issue at hand. Additionally validation of the api specification that works exactly the same as the one in APIM would be great to have at this point.

EDIT2: Add info about the wrong response status that is required so that the long running operation is taken care of

[aowrew]

Hi @daviian , I ran into this issue and tried different api versions but none worked for me. My specification is only ~200 LOC, yet more than 50% of operations failed to create. My API had revisions though - once I've removed the revisions from the code and re-deployed the API, the error went away and the API created successfully.