Improve entity discovery performance

[lacasseio]

trafficstars

The universal model migration put Nokee a bit behind Gradle core plugins in terms of performance. The migration of gradle/native-platform continues to show a ~900ms regression compared to the Software Model plugins. Despite the profiler doesn't point to an obvious bottleneck, we have a logical bottleneck regarding entity discovery. When entities are not required, we should avoid any sort of discovery or realization. At the moment, we mostly plan for discovery avoidance but never turned it on.

We did a small spike which was enabled by our recent cleanup work and we got the following numbers based on gradle/native-platform for up-to-date assemble:

With the base spike, we only reduce the discovery by a little bit. We discover 452 entities which take about 1400ms. Of those 452 entities, 152 entities are discovered just for the calculation of the variant buildability. By modifying the buildability calculation we can bring the total entities discovered to 300. Assuming the initial 1400ms is uniformly distributed among all entities, we could expect a performance of around ~930ms.

For on-par performance with the Software Model, it must take no more than 575ms which would roughly equal to discovering 137 entities. If we can avoid the discovery of all unbuildable variants we could bring the discovery total to about 112 entities which should be ~346ms. This would be below the Software Model performance by ~40%.

Those improvement could be undone depending on what the users do. For example, mapping providers successively can easily cause a lot of entities to be realized. We have some plans but nothing concrete that would prevent these scenario. We could open up the universal model but we fear a similar scenario as with the Software Model could happen. We have an idea around ModelStream that we toyed in the past which would sit in between the vanilla model and universal model, meaning it feels like normal Gradle but it directly act on the universal model. Finally, we could introduce a check that warns users when discovering a lot more entities then required to build the task graph. Usually that would mean a configuration problem. Regardless, the current focus is on the discovery avoidance work.

Step to the solution:

1. Finish the on demand entity discovery. There are some places in the code where we need to force children discovery. Some other places needs to rewriten to avoid discovering while we are discovering which should be illegal.
2. Rewrite buildability logic to avoid realizing shared library and its children. We can use some educated guess to make all the checks at the variant level avoiding discovery of the children.
3. Rewirte buildability logic for Ncurse variant in gradle/native-platform. The project requires additional logic which requires the realization of the variants. We will need to explore ways to avoid that.

In between each steps, we should see an improvement which the Gradle profiler will confirm.