Support to deploy all changes between two deployments instead of limiting it to last commit or entire repo content

[rohit3d2003]

Please describe the feature.

As per my understanding, the publisher currently supports two scenarios:

1. Deployment based on a single Commit ID
2. Deployment based on entire repo content

If a team is following GitHub Flow branching strategy and each environment based deployment is defined as 'Stages' within pipeline, how does a deployment in higher environment can support the delta commits between what was deployed last time and all the commits that happened afterwards. For example,

1. Developer made two separate commits and both commits are deployed in Dev environment. For both these runs, deployment to higher environment was cancelled
2. Now Developer made another commit, fixing an issue with last two commits and this commit should now be deployed to higher environment, does the last commit option automatically picks the delta between what's already deployed and what's new OR the only available option is to deploy entire repo content

[guythetechie]

Thanks for the feedback, @rohit3d2003. We currently don't support that scenario.

In your example, only changes in the last commit would be picked up. The previous commits would be skipped. Your only option to "true" things up would be to redeploy the entire repo content.

We didn't have great ideas on how to account for "missed" commits (other than a full deployment). Would love to hear any suggestions you may have.

[rohit3d2003]

Let me think through that and see what else can be done.

[waelkdouh]

@rohit3d2003 any update on this?

[rohit3d2003]

I couldn't think of anything else besides re-deploying the artifacts from repo all over again and below are the steps that could happen:

1. Create/Update the APIs based on the artifacts
2. Delete the APIs, Backends etc. that aren't present in artifacts folder to true things up. This will fix the scenario where an API should no longer be present in a higher environment if the same is deleted from repo

This can also be made optional by introducing a flag in publisher pipeline

[Gabriel123N]

@rohit3d2003 I face the same issue and this is pretty much the same solution I came up with. I am currently trying the following changes:

1. Add a new "PAST_COMMIT_ID" to define from where you want to monitor changes
2. Use git rev-list to get all the commits between the PAST_COMMIT_ID and COMMIT_ID
3. Enumerate through all the commits retrieved and check for deleted files
4. If an artifact was deleted at any point, delete it after going through all commits
5. Publish artifacts with the Publish all artifacts behaviour

But I don't have much time to work on it. However, if anything positive comes out of it I'll keep you updated and possibly make a PR.

[Gabriel123N]

@rohit3d2003 So I had the time to build a quick experimental mode in the following branch https://github.com/Gabriel123N/apiops/tree/delete-incremental . Here is how it works:

1. In the new mode publish-from-past-commit, you provide the full commit id in PAST_COMMIT_ID
2. Preferably, check in the IGNORE_VALIDATION check box. This allows you to ignore any 404 error that could happen when deleting products
3. The pipeline will build the modified code and run it instead of fetching the release on this repo
4. When the publisher is launched, it will retrieve the timeline between the current commit and the PAST_COMMIT_ID
5. Each commit will be checked out and delete artifacts as if you were running in publish-artifacts-in-last-commit
6. When all commits have been checked out, the publisher will run as if you choose publish-all-artifacts-in-repo mode

If this behaviour matches what you are expecting, feel free to pull the branch and run it against (and only against) a test APIM doing the following:

• Create a dummy branch where you will extract the new tools folder
• Run the publisher with publish-all-artifacts-in-repo against your test APIM
• Note the commit ID
• Do a few manual adjustments to your artifacts (Delete APIs, change a displayName here and there, add a new name value, ...) in ADO
• Run the publisher in publish-from-past-commit mode with the commit id your wrote above inside PAST_COMMIT_ID

[verlindb]

A temp workaround - apply squash merging on branches . We have achieved this by using tags that keep the state of the published environments. In our pipeline we use an intermediate branch for the rollout to apim instances. So we create a intermediate branch based on the tag and squash commit the changes from the CI triggerd branch. Next we get the latest commit and pass it to the publisher task.

This scenario will allow CI/CD for different stages. @waelkdouh

[martin2176]

@waelkdouh @guythetechie @verlindb @rohit3d2003 @Gabriel123N I have modified the GHA publisher pipeline of mine (experimentally) to ask for COMMIT_ID instead of inferring the commit_id based on last commitid. So if my prod instance missed 2 commits (say prod missed commitid A and B, if I run the published pipeline first with commitid:A and then rerun with commitid:B , wouldnt that rollout all the changes I missed? OR am I missing something in this logic?

[guythetechie]

@martin2176 - I think it would work. The publisher shouldn't care whether the specified commit ID was the last commit.

[waelkdouh]

Please describe the feature.

As per my understanding, the publisher currently supports two scenarios:

1. Deployment based on a single Commit ID

2. Deployment based on entire repo content

If a team is following GitHub Flow branching strategy and each environment based deployment is defined as 'Stages' within pipeline, how does a deployment in higher environment can support the delta commits between what was deployed last time and all the commits that happened afterwards. For example,

1. Developer made two separate commits and both commits are deployed in Dev environment. For both these runs, deployment to higher environment was cancelled

2. Now Developer made another commit, fixing an issue with last two commits and this commit should now be deployed to higher environment, does the last commit option automatically picks the delta between what's already deployed and what's new OR the only available option is to deploy entire repo content

Please remember to close the issue if you're satisfied that the resolution