[BUG] Publisher is stuck updating API when publishing 10+ APIs

[aimtiaz11]

Release version

v5.1.4 - Linux x64

Describe the bug

While running the publisher via Azure DevOps, we are publishing large number (12 or so) of APIs. The extractor tool seem to fire multiple HTTP requests to add/update them simultaneously and then gets stuck. The

Every Pipeline run, its stuck at a different APIs. From the debug logs, it seems to be stuck at PutRestResource[0] phase.

I am running via Azure DevOps pipeline with Ubuntu 22.0 Image. Using Linux x64 of the publisher.

Sample log below (with some values redacted).

info: Publisher[0]
      Putting API some-lookup-apiv1...
info: Publisher[0]
      Putting API petstore-apiv1...
info: Publisher[0]
      Putting API building-apiv1...
info: Publisher[0]
      Putting API petstore-apiv1;rev=1...
info: Publisher[0]
      Putting API some-taken-pr-apiv1...
info: Publisher[0]
      Putting API f-rating-xxx-anzov1...
info: Publisher[0]
      Putting API ofem-api...
info: Publisher[0]
      Putting API hdca-apiv1...
info: Publisher[0]
      Putting API sitcom-api-anzov1...
info: Publisher[0]
      Putting API digi-serv-xxx-anzov1...
info: Publisher[0]
      Putting API break-apiv1...
info: Publisher[0]
      Putting API some-lookup-apiv2...
info: Publisher[0]
      Putting API utitlity-some-api-apiv1...
dbug: PutRestResource[0]
      Beginning request to put REST resource URI https://management.azure.com/subscriptions/***/resourceGroups/some-rg/providers/Microsoft.ApiManagement/service/some-service/apis/some-lookup-apiv1?api-version=2022-04-01-preview...
dbug: PutRestResource[0]
      Beginning request to put REST resource URI https://management.azure.com/subscriptions/***/resourceGroups/some-rg/providers/Microsoft.ApiManagement/service/some-service/apis/petstore-apiv1;rev=1?api-version=2022-04-01-preview...
dbug: PutRestResource[0]
      Beginning request to put REST resource URI https://management.azure.com/subscriptions/***/resourceGroups/some-rg/providers/Microsoft.ApiManagement/service/some-service/apis/some-lookup-apiv2?api-version=2022-04-01-preview...
dbug: PutRestResource[0]
      Beginning request to put REST resource URI https://management.azure.com/subscriptions/***/resourceGroups/some-rg/providers/Microsoft.ApiManagement/service/some-service/apis/petstore-apiv1?api-version=2022-04-01-preview...
dbug: PutRestResource[0]
      Beginning request to put REST resource URI https://management.azure.com/subscriptions/***/resourceGroups/some-rg/providers/Microsoft.ApiManagement/service/some-service/apis/building-apiv1?api-version=2022-04-01-preview...
dbug: PutRestResource[0]
      Beginning request to put REST resource URI https://management.azure.com/subscriptions/***/resourceGroups/some-rg/providers/Microsoft.ApiManagement/service/some-service/apis/some-taken-pr-apiv1?api-version=2022-04-01-preview...
dbug: PutRestResource[0]
      Beginning request to put REST resource URI https://management.azure.com/subscriptions/***/resourceGroups/some-rg/providers/Microsoft.ApiManagement/service/some-service/apis/utitlity-some-api-apiv1?api-version=2022-04-01-preview...
dbug: PutRestResource[0]
      Beginning request to put REST resource URI https://management.azure.com/subscriptions/***/resourceGroups/some-rg/providers/Microsoft.ApiManagement/service/some-service/apis/hdca-apiv1?api-version=2022-04-01-preview...
dbug: PutRestResource[0]
      Beginning request to put REST resource URI https://management.azure.com/subscriptions/***/resourceGroups/some-rg/providers/Microsoft.ApiManagement/service/some-service/apis/x-rating-xxx-anzov1?api-version=2022-04-01-preview...
dbug: PutRestResource[0]
      Beginning request to put REST resource URI https://management.azure.com/subscriptions/***/resourceGroups/some-rg/providers/Microsoft.ApiManagement/service/some-service/apis/digi-serv-xxx-anzov1?api-version=2022-04-01-preview...
dbug: PutRestResource[0]
      Beginning request to put REST resource URI https://management.azure.com/subscriptions/***/resourceGroups/some-rg/providers/Microsoft.ApiManagement/service/some-service/apis/comm-apiv1?api-version=2022-04-01-preview...
dbug: PutRestResource[0]
      Beginning request to put REST resource URI https://management.azure.com/subscriptions/***/resourceGroups/some-rg/providers/Microsoft.ApiManagement/service/some-service/apis/ofem-api?api-version=2022-04-01-preview...
dbug: PutRestResource[0]
      Beginning request to put REST resource URI https://management.azure.com/subscriptions/***/resourceGroups/some-rg/providers/Microsoft.ApiManagement/service/some-service/apis/sitcom-api-anzov1?api-version=2022-04-01-preview...
dbug: PutRestResource[0]
      Beginning request to put REST resource URI https://management.azure.com/subscriptions/***/resourceGroups/some-rg/providers/Microsoft.ApiManagement/service/some-service/apis/break-apiv1?api-version=2022-04-01-preview...
dbug: PutRestResource[0]
      Successfully put REST resource at URI https://management.azure.com/subscriptions/***/resourceGroups/some-rg/providers/Microsoft.ApiManagement/service/some-service/apis/petstore-apiv1?api-version=2022-04-01-preview.
dbug: PutRestResource[0]
      Successfully put REST resource at URI https://management.azure.com/subscriptions/***/resourceGroups/some-rg/providers/Microsoft.ApiManagement/service/some-service/apis/utitlity-some-api-apiv1?api-version=2022-04-01-preview.
dbug: PutRestResource[0]
      Successfully put REST resource at URI https://management.azure.com/subscriptions/***/resourceGroups/some-rg/providers/Microsoft.ApiManagement/service/some-service/apis/some-lookup-apiv2?api-version=2022-04-01-preview.
dbug: PutRestResource[0]
      Successfully put REST resource at URI https://management.azure.com/subscriptions/***/resourceGroups/some-rg/providers/Microsoft.ApiManagement/service/some-service/apis/some-lookup-apiv1?api-version=2022-04-01-preview.
dbug: PutRestResource[0]
      Successfully put REST resource at URI https://management.azure.com/subscriptions/***/resourceGroups/some-rg/providers/Microsoft.ApiManagement/service/some-service/apis/f-rating-xxx-anzov1?api-version=2022-04-01-preview.
dbug: PutRestResource[0]
      Successfully put REST resource at URI https://management.azure.com/subscriptions/***/resourceGroups/some-rg/providers/Microsoft.ApiManagement/service/some-service/apis/digi-serv-xxx-anzov1?api-version=2022-04-01-preview.
dbug: PutRestResource[0]
      Successfully put REST resource at URI https://management.azure.com/subscriptions/***/resourceGroups/some-rg/providers/Microsoft.ApiManagement/service/some-service/apis/building-apiv1?api-version=2022-04-01-preview.
dbug: PutRestResource[0]
      Successfully put REST resource at URI https://management.azure.com/subscriptions/***/resourceGroups/some-rg/providers/Microsoft.ApiManagement/service/some-service/apis/some-taken-pr-apiv1?api-version=2022-04-01-preview.
      ```

### Expected behavior

Publisher should update all APIs

### Actual behavior

Publisher fires multiple REST API calls (PutRestResource) and gets stuck with no further movement.

### Reproduction Steps

Requires at least 10 APIM APIs. Extract to Branch, and then publish to same or different APIM instance.

[github-actions[bot]]

  Thank you for opening this issue! Please be patient while we will look into it and get back to you as this is an open source project. In the meantime make sure you take a look at the [closed issues](https://github.com/Azure/apiops/issues?q=is%3Aissue+is%3Aclosed) in case your question has already been answered. Don't forget to provide any additional information if needed (e.g. scrubbed logs, detailed feature requests,etc.).
  Whenever it's feasible, please don't hesitate to send a Pull Request (PR) our way. We'd greatly appreciate it, and we'll gladly assess and incorporate your changes.

[mattiabonanni]

We're having the same issue, @aimtiaz11 did you manage to get around the problem?

[waelkdouh]

Are you using the latest v6 rc version?

[mattiabonanni]

As for us, the latest stable release v5.1.4. Is it a known bug which was fixed?

[waelkdouh]

As for us, the latest stable release v5.1.4. Is it a known bug which was fixed?

@guythetechie thoughts on this?

In the meantime @MattiaBonanni if you can try v6 it would be great. It's a major shift in how we store the artifacts in v5 not to mention several bug fixes. So I highly recommend upgrading to v6 asap. We will also be adding workspaces support under v6 soon. Let me know how that goes.

[mattiabonanni]

Ok thanks, I'll try it out as soon as I can

[aimtiaz11]

I can confirm this issue is resolved in v6.0.0-rc1. Thanks @waelkdouh.

[mattiabonanni]

@waelkdouh we tried to update it, but we've got the same error. Is there some debug logs which can be enabled to check if there's more information?

[waelkdouh]

Yes. Please check the wiki section for the variable you can set to enable logging. Of course you have access to the source code if you want to modify it and rebuild it. We are always open to PRs.