Parallel build support

[mikezerosix]

We have a lot of proxies and running deploy takes looooong time.

We structured our solution as Maven multi module project. So simply running from root all we can easily deploy them all and we also can easily choose to deploy them individually from each submodule.

But would be great if we would deploy all of them at once using Maven parallel build: mvn -T 10 install (install phase runs the apigee plugin)

We have tried parallel build, but it fail with strange errors like :

[ERROR] Failed to execute goal io.apigee.build-tools.enterprise4g:apigee-edge-maven-plugin:2.5.0:deploy (deploy-bundle) on project shared-request-player: MojoExecutionException: com.google.api.client.http.HttpResponseException: 404 Not Found
[ERROR] {
[ERROR]   "error": {
[ERROR]     "code": 404,
[ERROR]     "message": "generic::not_found: organizations/xxx/sharedflows/shared-xxx/revisions/218 not found",
[ERROR]     "status": "NOT_FOUND",
[ERROR]     "details": [
[ERROR]       {
[ERROR]         "@type": "type.googleapis.com/google.rpc.RequestInfo",
[ERROR]         "requestId": "11261845108213792315"
[ERROR]       }
[ERROR]     ]
[ERROR]   }
[ERROR] }

That shared flow has no dependencies and it deploys without any issues in non-parallel build. Cutting down the half hour deploy time to just 3 minutes being able to run it in parallel would be amazing.

So if possible, please suppport Maven parallel builds.

[ssvaidyanathan]

@mikezerosix - I believe you just want to perform fire and forget style deployments and not wait for the deployments to complete. The default option is that so that you can include triggering the automated testing after knowing that deployment is complete.

If you do not want that - you can change the deploy options. Just use -Dapigee.options=async or update the apigee.options value in the pom profile to <apigee.options>async</apigee.options>

That will just run the import and deploy command and not poll for the deployment to complete. The same is documented as well

[mikezerosix]

@ssvaidyanathan Thanks. I am aware of the async option. But I do want to wait for confirmation, I just do not run deployments sequentially. I have maven multimodule project with 20 sub modules, each is apigee flow or proxy. if I run the maven normally mvn install it takes maven almost an hour to run the build as it runs each proxy deployment one at time wating for one to complete before running the next. What I do want to, is to run Maven with mvn -T 20 which would run maven in 20 threads, running all 20 submodules at once and it would complete in 2 minutes (time of single polled deployment) But when I try to run maven in parallel (even just "-T 2", cutting build time to half), I get errors from this plugin.

[ERROR] Failed to execute goal io.apigee.build-tools.enterprise4g:apigee-edge-maven-plugin:2.5.0:deploy (deploy-bundle) on project shared-request-player: MojoExecutionException: com.google.api.client.http.HttpResponseException: 404 Not Found
[ERROR] {
[ERROR]   "error": {
[ERROR]     "code": 404,
[ERROR]     "message": "generic::not_found: organizations/xxx/sharedflows/shared-xxx/revisions/218 not found",

So it looks like this plugin does not support multithreading and it would be amazing if it did.

Or I would love have ability to run async deploy and then run separate process to poll projects deployment

1. First run mvn apigee-edge-maven-plugin:deploy -Dapigee.options=async Which would run the 20 sub modules each with proxy in less than minute.

2. Then run mvn apigee-edge-maven-plugin:POLL-ASYNC-DEPLOYMENTS Which would poll and untill each of the previously deployed proxies and shared flows report having been deployes or report failure. This should complete in 2-3 minutes assuming Apigee itself runs deployments paraller.

Problem here would be that this would need to persist information between these two runs (like mvn release plugin) so it knows which revisions to poll for. So it does not get confused from 3rd party deploys.

[ssvaidyanathan]

@mikezerosix - Thank you for the detailed summary. Great pointers and suggestions. Let me see what can be done. Will try them out. This is the first time someone's asking this. So thats great 😄

I will need a favor - if I end up implementing, I will need you to help test it for me and see if it works as expected. We can fine tune as you provide more inputs.

[mikezerosix]

I created sample project for the issue. I have not tested it that it actually runs, but it at least demonstrates the structure https://gitlab.com/mike06/public/apigee-paraller-deploy-sample

Failure is random, but on thread value 4 or higher it should be almost 100%.

[ssvaidyanathan]

Brilliant.. I will use this to test it out

[mikezerosix]

I updated the sample little bit. I also added the Maven warning about threads

[WARNING] ***************************************************************** [WARNING] * Your build is requesting parallel execution, but this * [WARNING] * project contains the following plugin(s) that have goals not * [WARNING] * marked as thread-safe to support parallel execution. * [WARNING] * While this /may/ work fine, please look for plugin updates * [WARNING] * and/or request plugins be made thread-safe. * [WARNING] * If reporting an issue, report it against the plugin in * [WARNING] * question, not against Apache Maven. * [WARNING] ***************************************************************** [WARNING] The following plugins are not marked as thread-safe in proxynn: [WARNING] io.apigee.build-tools.enterprise4g:apigee-edge-maven-plugin:2.5.0 [WARNING] [WARNING] Enable debug to see precisely which goals are not marked as thread-safe. [WARNING] *****************************************************************

[ssvaidyanathan]

@mikezerosix - I cloned your repo I had to make some changes for example

• add <apigee.apiversion>v1</apigee.apiversion> to main pom.xml
• add <module>proxy18</module> to the list of modules in the main pom.xml
• update the <BasePath> in Proxy04 module.
• remove the apigee.googletoken.email and oauthClientId from the profile in the main pom.xml

I ran the test with -T4C and also with -T10C, I saw errors popping out mainly 503s from the Apigee APIs (I think its rate limiting) when the plugin is polling for the Deployment state

However, when I ran the same test with <apigee.options>async</apigee.options>, it worked without any issues

The entire build ran in 10 seconds

[INFO] Reactor Summary for apigee-maven-parallel 0.1.0-SNAPSHOT:
[INFO] 
[INFO] apigee-maven-parallel .............................. SUCCESS [  0.910 s]
[INFO] proxy01 ............................................ SUCCESS [  4.447 s]
[INFO] proxy02 ............................................ SUCCESS [  4.220 s]
[INFO] proxy03 ............................................ SUCCESS [  8.624 s]
[INFO] proxy04 ............................................ SUCCESS [  5.868 s]
[INFO] proxy05 ............................................ SUCCESS [  7.956 s]
[INFO] proxy06 ............................................ SUCCESS [  3.782 s]
[INFO] proxy07 ............................................ SUCCESS [  5.510 s]
[INFO] proxy08 ............................................ SUCCESS [  7.240 s]
[INFO] proxy09 ............................................ SUCCESS [  4.837 s]
[INFO] proxy10 ............................................ SUCCESS [  3.058 s]
[INFO] proxy11 ............................................ SUCCESS [  5.210 s]
[INFO] proxy12 ............................................ SUCCESS [  6.941 s]
[INFO] proxy13 ............................................ SUCCESS [  6.275 s]
[INFO] proxy14 ............................................ SUCCESS [  7.607 s]
[INFO] proxy15 ............................................ SUCCESS [  6.581 s]
[INFO] proxy16 ............................................ SUCCESS [  9.361 s]
[INFO] proxy17 ............................................ SUCCESS [  2.542 s]
[INFO] proxy18 ............................................ SUCCESS [  3.417 s]
[INFO] proxy19 ............................................ SUCCESS [  8.312 s]
[INFO] proxy20 ............................................ SUCCESS [  8.954 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  10.383 s (Wall Clock)
[INFO] Finished at: 2024-05-29T17:52:04-07:00
[INFO] ------------------------------------------------------------------------

I think you can ignore the threadsafe warning. Thats not really impacting the plugin.

With the default "override" option, the plugin is polling the deployments API and if the response from the Apigee API is 503, then the plugin just exits and marks the build as Failed.. But the proxy is eventually getting deployed.

What I tried is pipe the mvn response to a log file

mvn clean install -P destination1 -Dbearer="$(gcloud auth print-access-token)" -T10C > a.log

Then looked at the log in detail

[mikezerosix]

@ssvaidyanathan I added your fixes to repo. Thanks.

I forgot to mention that parallel deploy run might not fail on empty apigee. Because errors were about revisions. When the deploy actually overrides the existing revisions, it should fail. I have once successfull ran maven parallel with -T1C on 2 core cpu for 16 proxies and flows. But that could just been because the other core was buzy and never ran the second thread. As re-running it failed. Also most times error seems to come from flows, but proxies do fail also. If you can not reproduce the problem with this sample, I can add flows to it.

I know that warning is not critical, I added it just for information.

I would be interested to know how long does that take to run in default override option, in single thread. It should be something like 40 minutes, even though the proxies have no policies. I have no Apigee that I can run this sample myself as I do not want to mess around in customer's enviroment.

We tried asýnc option and it run the project without problems. But that async does not meet our requirements. We need to assert the proxies are successfully deployed. Using async the solution would be just horrible; unreliable and a lot of work to create and maintain
If we used async, we would

1. we would need parse the list of proxies that are supposed to be deployes. (parse the pom.xml at deploy time to prevent human error.
2. query currently deployed revision of each of those proxies
3. store this information for later
4. run deploy async
5. create ad hoc poller to check all those proxies and flows for status and deployed revision.
6. compare these to stored revisions
7. hope and pray one one else deloys any of those proxies while this one hour long process is running !

We could do the work, but the fact that it would be unreliable is a deal breaker.

[ssvaidyanathan]

Got it.. The issue with "override" is that the Apigee APIs are rate limiting.. So you really cannot do anything in the plugin. I tried the async multiple times so that new revisions of the proxy gets created. It always worked.

But I was always having issues with "override" since the Apigee APIs were rate limiting and that was causing failures. Dont think adding flows or policies would matter here.

Let me think about your mvn apigee-edge-maven-plugin:POLL-ASYNC-DEPLOYMENTS option. The issue here is - its the same set of GET API that I will have to call for polling which is used with the "override". So not sure how this will help.

[mikezerosix]

@ssvaidyanathan About ratelimit, what about adding plugin option for poll rate ? At what rate does the plugin now poll Apigee ? Does Apigee give specific error on rate limit exceeded, so could pluging throttle & wait when receiving rate limit exceed error and resume polling ?

[mikezerosix]

There is a throttling trick I have used in polling, instead of running immediately and sleeping for pollRate. I start the poll on even clock seconds of the pollRate. Setting different projects pollRate to prime numbers, you can get five ticks that never tick at same time. For example is I would set 4 sub modules <apigee.poll.rate>31</apigee.poll.rate> another 4 to <apigee.poll.rate>33</apigee.poll.rate>, another 4 to <apigee.poll.rate>35</apigee.poll.rate> and so on; 37, 39

In Java language: Instead of doing sleet(31) ... Might cause all pols to tick at same time even if rate is different

But doing: cronSchedule("0/31 * * * * ?") ...

Guarantees that with 20 divided to prime pollRates the parallel polls to only ever run 4 polls per 3 seconds and 20 polls a minute.

[ssvaidyanathan]

@mikezerosix - did u close this issue by mistake?

[mikezerosix]

yes, I closed by mistake

[ssvaidyanathan]

@mikezerosix - when I run your setup, I only see 503s and no 404s Are you getting both 503s and 404s? or just 404s?

[ssvaidyanathan]

@mikezerosix - Just released v2.5.1 Can you update your pom to point to the above version and re-run your cases. I ran it a few times and it worked for me

[mikezerosix]

@ssvaidyanathan I was consistently getting 404 error when they plugin tried to retrieve the revisions for sharedflow or proxy.

I updated the Gitlab sample to 2.5.1 and I updated the real project to 2.5.1. I tried to run the real project with plugin 2.5.1 and mvn run parameter -T5C, which for the 2 core runner machine would have tried to run 2*5=10 threads. That gave the same 404 erro on trying to retrieve some proxy's revisions.

Running with -T1C that should run 21=2 threads succeeded and reduced total runtime by half as expected. Running with -T2C that should run 22=4 threads failed on last sharedflow with 404 on revisions. Otherwise that would had reduced the total runtime by factor of 4.

I only ran -T1C (=2 threads) once, so it might not be stable. I have once succeeded running that o version 2.5.0, which succeeds something like 1 out of 10 times.

[mikezerosix]

On both 2.5.0 and 2.5.1 errors are always something like

[ERROR] Failed to execute goal io.apigee.build-tools.enterprise4g:apigee-edge-maven-plugin:2.5.0:deploy (deploy-bundle) on project shared-request-player: MojoExecutionException: com.google.api.client.http.HttpResponseException: 404 Not Found
[ERROR] {
[ERROR]   "error": {
[ERROR]     "code": 404,
[ERROR]     "message": "generic::not_found: organizations/xxx/sharedflows/shared-xxx/revisions/218 not found",

Either on sharedflow or proxy.

[ssvaidyanathan]

@mikezerosix - I never got a 404 Can you try running on the GitLab code you shared. We can test using that set of proxies

[mikezerosix]

@ssvaidyanathan Sorry, like I said earlier, I can not run the sample on Apigee. As Apigee instances require (very expensive) license I do not have playground Apigee instance. I only have real customer Apigee instance and I can not clutter that with the sample. I ran the real customer project with 20 mixed sharedflows and proxies using plugin v2.5.1 to the Apigee with above results.

[ssvaidyanathan]

@mikezerosix - then its difficult for me to fix this issue. From my tests, its working as expected as it pushed all the proxies multiple times. Below are some file runs that worked for me debug3.log debug5.log

[mikezerosix]

@ssvaidyanathan I got permission to run my sample in customer's Apigee. I let you know the result when I can run it.

[mikezerosix]

@ssvaidyanathan Amazing work. It works like charm. Can you do same fix for sharedflows too now ?

Result:

1. Baseline - Running 20 dummy proxies with maven plugin with polling of deployment to complete (apigee.options=override) took:

root .................................. SUCCESS [ 3.205 s]
proxy01 ............................................ SUCCESS [02:23 min]
proxy02 ............................................ SUCCESS [01:31 min]
proxy03 ............................................ SUCCESS [01:30 min]
proxy04 ............................................ SUCCESS [01:30 min]
proxy05 ............................................ SUCCESS [01:29 min]
proxy06 ............................................ SUCCESS [01:29 min]
proxy07 ............................................ SUCCESS [01:29 min]
proxy08 ............................................ SUCCESS [01:30 min]
proxy09 ............................................ SUCCESS [01:31 min]
proxy10 ............................................ SUCCESS [01:30 min]
proxy11 ............................................ SUCCESS [01:29 min]
proxy12 ............................................ SUCCESS [01:28 min]
proxy13 ............................................ SUCCESS [01:31 min]
proxy14 ............................................ SUCCESS [01:29 min]
proxy15 ............................................ SUCCESS [01:29 min]
proxy16 ............................................ SUCCESS [01:29 min]
proxy17 ............................................ SUCCESS [01:29 min]
proxy18 ............................................ SUCCESS [01:30 min]
proxy19 ............................................ SUCCESS [01:30 min]
proxy20 ............................................ SUCCESS [01:29 min]
------------------------------------------------------------------------
BUILD SUCCESS
------------------------------------------------------------------------
Total time: 31:01 min

2. Asyncronious deploy where result is unkown making this option unusable. When running with Maven without polling for completion (apigee.options=async) took:

root .................................. SUCCESS [ 3.058 s]
proxy01 ............................................ SUCCESS [ 2.850 s]
proxy02 ............................................ SUCCESS [ 1.587 s]
proxy03 ............................................ SUCCESS [ 2.319 s]
proxy04 ............................................ SUCCESS [ 2.175 s]
proxy05 ............................................ SUCCESS [ 2.225 s]
proxy06 ............................................ SUCCESS [ 1.538 s]
proxy07 ............................................ SUCCESS [ 1.526 s]
proxy08 ............................................ SUCCESS [ 1.491 s]
proxy09 ............................................ SUCCESS [ 1.617 s]
proxy10 ............................................ SUCCESS [ 1.608 s]
proxy11 ............................................ SUCCESS [ 2.273 s]
proxy12 ............................................ SUCCESS [ 2.255 s]
proxy13 ............................................ SUCCESS [ 1.520 s]
proxy14 ............................................ SUCCESS [ 2.240 s]
proxy15 ............................................ SUCCESS [ 2.857 s]
proxy16 ............................................ SUCCESS [ 1.516 s]
proxy17 ............................................ SUCCESS [ 2.241 s]
proxy18 ............................................ SUCCESS [ 2.843 s]
proxy19 ............................................ SUCCESS [ 1.696 s]
proxy20 ............................................ SUCCESS [ 2.916 s]
------------------------------------------------------------------------
BUILD SUCCESS
------------------------------------------------------------------------
Total time: 44.519 s

3. Real test - Running 20 dummy proxies with maven plugin with polling of deployment to complete (apigee.options=override) and maven parallel threads (-T5C) took:

[INFO] root .................................. SUCCESS [ 3.111 s]
proxy01 ............................................ SUCCESS [ 50.535 s]
proxy02 ............................................ SUCCESS [ 48.935 s]
proxy03 ............................................ SUCCESS [01:52 min]
proxy04 ............................................ SUCCESS [ 48.934 s]
proxy05 ............................................ SUCCESS [01:52 min]
proxy06 ............................................ SUCCESS [01:52 min]
proxy07 ............................................ SUCCESS [ 50.268 s]
proxy08 ............................................ SUCCESS [ 48.627 s]
proxy09 ............................................ SUCCESS [01:53 min]
proxy10 ............................................ SUCCESS [ 58.232 s]
proxy11 ............................................ SUCCESS [01:10 min]
proxy12 ............................................ SUCCESS [01:08 min]
proxy13 ............................................ SUCCESS [01:10 min]
proxy14 ............................................ SUCCESS [ 59.356 s]
proxy15 ............................................ SUCCESS [01:00 min]
proxy16 ............................................ SUCCESS [ 59.079 s]
proxy17 ............................................ SUCCESS [01:08 min]
proxy18 ............................................ SUCCESS [ 58.854 s]
proxy19 ............................................ SUCCESS [ 57.502 s]
proxy20 ............................................ SUCCESS [ 57.626 s]
------------------------------------------------------------------------
BUILD SUCCESS
------------------------------------------------------------------------
Total time: 03:01 min (Wall Clock)

Note the total on wall clock as those deploys are are paraller.

[mikezerosix]

@ssvaidyanathan And because why not ? Running -T10C also succeed and in 2 minutes wall clock. Again, amazing work, thank you. I hope you can add this fix to work with sharedflows too. As in real project is mix of shared flows and proxies.

[ssvaidyanathan]

@mikezerosix - Thats great news. Thanks for testing them out. Appreciate it.

Just to be sure,

1. Your Test 1, is deploying the proxies without the -T arg, correct? Thats why it took 30+ minutes.
2. The option should be "override" and not "overwrite". It doesnt matter as it defaults to "override"

The code change should include the change for sharedflows as well. As its the same function for both. Just that you need to pass <apigee.apitype>sharedflow</apigee.apitype> in the Maven profile

[mikezerosix]

@ssvaidyanathan

1. Yes, the first run was the the normal sequential run without "-T" for baseline runtime. And to confirm the empty dummy proxies take same time to deploy as real proxies with policies.
2. That was typo in my message, in code option has "override", I fixed the message text.

I added 10 sharedflows to the Sample project. They seem to working. I might add another 10 later when I have more time. But I think 10 + 20 is pretty good sample.

[ssvaidyanathan]

Thats awesome!! Keep me posted. We can close this issue after your tests.

[mikezerosix]

@ssvaidyanathan Unfortunately the real project still fails. Still with 404 and it is using io.apigee.build-tools.enterprise4g:apigee-edge-maven-plugin:2.5.1:deploy

There should not be any other difference between the real project and sample than that real project had more content on the proxies and shared flows than the sample.

Real project always fails on sharedflows, never on proxies. Were both proxies and sharedflows supposed to be fixed for parallel run in plugin version 2.5.1 or was it just proxies ? And were you going to do yet another release where you add the same fix to sharedflows deploy too ?

[ssvaidyanathan]

No - no other change needed for sharedflows. If you are sure that its failing only for sharedflows, can you confirm that for those modules if <apigee.apitype>sharedflow</apigee.apitype> is set correctly

[mikezerosix]

It is and project deploys fine sequentially, without -T mvn run parameter.

[ssvaidyanathan]

Any chance you can share the log file?