Odd behavior when tearing down VMs at end of scale_out test

[pdmazz]

At the completion of the scale out test, all the AIs and VMs are destroyed. I have set the detach_parallelism to 3 for vm_defaults, aidrs_defaults, and ai_defaults in the osgcloud_rules.yaml file.

The tear downs will begin with 3 AIs and their associated VMs. When an AI-XXX is terminated/made undefined, another AI-YYY will begin the tear down process.

However, several times I have seen odd pauses in the tear down.

For example, there are three AIs in the process of being terminated: AI-2, AI-4, and AI-8.

The order of completion of termination is: AI-4, AI-8, then AI-2. Though AI-4 and AI-8 have terminated, no other AI terminates are initiated.

After AI-2 completes, it detects that it was successfully undefined and begins termination of AI-10. It did NOT spawn any other terminations [AI-1, AI-11, and AI-12 are waiting to be terminated].

AI-10 completes and it detected as successfully undefined and it now detects that AI-4 and AI-8 have completed/undefined [even though they completed several minutes prior]. AI-11, AI-12, and AI-1 terminations are spawned.

[mraygalaxy]

Greetings @pdmazz Some

questions:

1. Did all of the terminations eventually fully complete?
2. What cloud are you using?
3. If you do $ grep ERROR /var/log/cloudbench/XXXX_operations.log, do you find any meaningful errors between cloudbench and the interations with your cloud provider?
4. How many AIs in total are we talking about?

[pdmazz]

Michael, sorry for the tardy reply, the GitHub email notification went into my junk folder :-(

1. Yes, everything eventually terminated successfully.
2. I am using IBM Cloud (SoftLayer).
3. There were 13 AIs in total.
4. At the point of the tear down, there was this single message in the cbuser_operations.log file:

Oct 23 03:29:06 pdmazz-spec2-mel01.ibmperf.com cloudbench [ERROR] base_operations.py/ActiveObjectOperations.initialize_object TEST_cbuser - AI object initialization failure: need to pass through the appropriate 'all' function

[mraygalaxy]

I suspect something is going on with the softlayer adapter here. @maugustosilva comments?