actonc should build modules concurrently

[plajjan]

Would be cool if we could do concurrent builds. Not super important since we don't have any projects with many modules and even if we did, compilation is currently quite fast anyway.

Anyway, we spoke very briefly about it in today's meeting. While figuring out dependencies and ordering is quite straight forward, @nordlander mentioned challenges of serializing IO, since like updating the archives must not happen concurrently.

I suppose we could do as many threads as we have CPU cores per default.

[plajjan]

I found https://hackage.haskell.org/package/pooled-io which seems to be made for exactly this kind of use, in particular InOrder https://hackage.haskell.org/package/pooled-io-0.0.2.2/docs/Control-Concurrent-PooledIO-InOrder.html seems like it should be a very good fit as it supports dependencies.

As for IO, can't we just wrap that particular part, i.e. running ar under an internal lock? I don't think there is any other activity that would need to be serialized/locked.

[plajjan]

@sydow I wonder, is this something you could take a look at?

It seems to me that it's fairly simple to use the above referenced library for concurrent execution while respecting dependencies... however, it's still Haskell so I'm struggling to apply it to our code, like how to map our module dependencies as the NFdata dependencies that pooled-io uses?. Also not sure if there are other dealbreakers, like it doesn't appear to deal well with exceptions. Is that a blocker for us perhaps?

I don't think concurrent compilation is really a high priority, but I figure if we can spend an hour or two on it and get it working, it would be very cool to have.

Also, if we think it complicates our code in such a way that it will be significantly harder to maintain, we should perhaps not do it either... thoughts? opinions? :)

[plajjan]

FWIW, I did spend some time a) building a test case with many acton modules, which could run much faster with concurrent compilation and b) added a lock around archive handling, since that is otherwise not concurrency-safe. I can try to clean up that code (it's old) and submit...

[mzagozen]

This would come in handy for compiling respnet with its many large dataclasses modules for service layers and devices. In that particular case there are no dependencies between the dataclasses modules either.

[plajjan]

Yeah, it would come in handy. My old comment above is irrelevant now. It related to how we previously compiled C files and constructed archives out of those. We have now delegated all of that to Zig build

• I think this will happen by letting acton act as a top level orchestrator that compiles invididual files by executing actonc on the specific file
• acton is written in Acton, so it's quite natural to work with this sort of reactive and concurrent program whereas actonc is in Haskell and it is IMHO less natural to express in Haskell
• actonc can stay single threaded (it is not a threaded program today) - simple etc, compilation of individual modules is sequential anyway
• actonc takes a lock, so we need to have an option to run lock-less and only allow top acton process to grab the lock and the underlying actonc processes to trust - no need for more fine-grained locks IMHO
  • [x] actonc --sub to tell it is a sub-compiler and thus does not need locks (maybe other things can be implied by --sub??)
• to orchestrate the compilation, acton needs a view of all dependencies / imports - need to run a first discovery phase to list all imports
  • [x] actonc needs option to list imports
• [ ] rest is just code

[plajjan]

Just as a comment on my above note - the concurrency is now implemented entirely in actonc with threads. We get close to linear scaling, so GC et al overhead seems acceptable and this is of course simpler in many ways than using acton.act as the overarching orchestrator to build multiple modules. I reckon we'll actually continue along these lines and try to do more within the same actonc process now that it's been proven to work (and after the GC tweak),