MobileNativeFoundation
Metric proposal: Calculate the simulator tracks' idle time
(Edited) Problem Statement: The numSims configuration indicates the maximum number of active simulators allowed. Imagine them as channels/tracks to accommodate the launched simulators. There may be times where the number of simulators is less than maximum, before/between simulator launches and towards the end when there are no test bundles to pick up while some simulators are still testing. This simulator track idle time is not being tracked.
Proposal: Assuming there are 'n' simulator tracks waiting to get stuff done right from when the Bluepill starts... it can measure and report the tracks' processing time that wasn't used. This metric should include the overhead of upfront computations (formation, splitting and/or ordering of test bundles, etc.,) and all the fragmentation towards the end of test execution where one or more tracks are idle, having no more test bundles to take up, while others are still busy.
Benefits: This would help us measure and compare multiple approaches (test-count based packing vs test-execution-time based packing of test bundles as discussed in https://github.com/linkedin/bluepill/issues/336) and identify when performance regressions happen.
As of now, there should be no simulator idle time. That is, Bluepill will launch a simulator with the specific purpose of running a test bundle, run it, then shut it down, and then delete it. There should be no idle simulators ever. I suspect what you're seeing are simulators in the process of shutting down or booting up. The trace-profile file should already track all the other overhead (and if something is missing, it should be easy to fix).
Or am I misunderstanding your proposal?
@ob, You are right. The simulators are not idle. It's the simulator channel/track that is idle some of the time. Let me elaborate.
Right now the numSims configuration indicates the maximum number of simulators that can be active at any point of time and it is not actually the number of simulators Bluepill will create. Bluepill treats numSims as the number of simulator channels/tracks that are available and creates simulators in those tracks as needed. So, what I am proposing is more of a simulator track idle time.
Imagine a scenario where 12 out of 13 test bundles are tested with 4 parallel simulators but the simulator that is testing the 13th bundle is only 10% done. While that simulator tests remaining 90% other tracks have nothing else to do and are idle. Now if the testing of that 13th bundle was started earlier and all 4 tracks are done about the same time, then the track idle time would have been smaller. That's what the metric is supposed to indicate.
Ah, that makes sense. You are talking about something like "lost opportunity of parallelizing".
I think this would need the same information as #336... namely the test times in order to organize the ordering of runs more efficiently.
@ravimandala is this still a valid problem? @ob were you suggesting to re-organize the trace profile per bp process?
