build
build copied to clipboard
dart-lang
Discuss: Use Isolates to parallelize builds
We currently only use a single core for builds. We have an understanding of things that could be done in parallel and spawning Isolates might help some builds.
This would be challenging for two reasons:
- The Analyzer would hurt performance unless we come up with a way for different Isolates to cooperate on analysis. This is most likely a blocker for this change at all, but we might be able to revisit if we can move to a different model for code resolution.
- We'd need fine-grained communication across Isolates since the unit we can parallelize at is per-asset/builder combo.
Filing to track possible designs and performance impacts, we do not have immediate plans to work on this.
There are other aspects to consider here as well, such as asset caching and general memory consumption impact.
I think we could probably have an asset reader which reads by communicating with the main isolate to limit memory usage but still get some benefit from caching, but we would have to test how fast that is though.
I think we could probably have an asset reader which reads by communicating with the main isolate
Yeah that's what I was thinking. If we ever end up needing to put our own interface on top of the analyzer model and we're able to design it such that we can serialize it across send ports that could also solve the cooperation for the resolver.
Yeah that's what I was thinking. If we ever end up needing to put our own interface on top of the analyzer model and we're able to design it such that we can serialize it across send ports that could also solve the cooperation for the resolver.
Or if we end up with a summary based analyzer api that would also work better with this model.
Another approach here would be to have angular actually run as a bazel worker, or even add generic support for running builders as bazel workers. We could add an isolate based bazel worker implementation as well.
I think we essentially have to have a summary-based analyzer implementation for this to be feasible - but that could have other benefits anyways. We are currently in the process of migrating to the new analyzer apis so this could happen after that.
Note that a summary based analyzer will also help a ton in other ways - specifically by reducing the number of dependencies for each action. The asset graph is extremely inefficient today because of how we use the analyzer (each output depends on all transitive Dart dependencies of its primary input).
Or, we could add support for only non-analyzer based builders or something, but those are generally very cheap already.
My understanding is that we do use analyzer summaries now, so I'm wondering if this is something that could be reconsidered?
I think builders would have to opt-in to this either way (the behavior is potentially breaking because two builders on different isolates can't share build resources), so maybe we can first start without analyzer support too. In my case, I have a set of builders that don't use the analyzer, are sort-of expensive and run on many assets. So I'm interested in parallel builds, which I think would improve performance here.
My understanding is that we do use analyzer summaries now, so I'm wondering if this is something that could be reconsidered?
We still don't use analyzer summaries actually
In my case, I have a set of builders that don't use the analyzer, are sort-of expensive and run on many assets. So I'm interested in parallel builds, which I think would improve performance here.
Fwiw, you could potentially use your own isolate, which you use a Resource to manage. That isn't all that different from how we use the compilers today. If you wanted to experiment with this that is probably where I would start.
Since isolates changed quite a bit recently and the repo as well, is it something now that would be more possible ? I think it might be a good way to improve performance ?
We still can't share an analysis context across isolates without fully copying it over afaik - its a mutable data structure and can't be shared.