HELICS
HELICS copied to clipboard
GMLC-TDC
Usage of HELICS, Few big federates vs many tiny ones.
HELICS was designed and build around the idea of bigger federates with lots of IO, each using multiple cores already and 1 maybe 2 on a single machine. Which is probably true for things like GridDyn, GridlabD, NS3, and other bigger tools.
So it is fully multithreaded, supports ASync IO processing, uses a number of threads handle the comms, and message processing, all to minimize latency of the actual Federate operations.
That said what I am hearing repeated is that a more common use case might actually be a tiny federate with very limited communications, running on a single thread and wanted a bunch (1000's in some cases) running on a single machine in independent processes. HELICS is not optimized for this case very well.
So my question is how valuable will it be longer term to make an option inside HELICS for a single threaded core. Strip out all multithreading in the federate and core and only support a single federate in a core.
Hmm, certainly want to keep the support for the higher performance, multi-threaded support since the large federates are important in many situations. That said, I agree that many tiny ones is a valid setup, and might actually be commonly used in conjunction with multiple larger ones too (e.g. T&D with lots of controllers).
Question to me is that it seems maintaining both single and multi-threaded could become a pain. How bad of performance hit would you expect using the multi-threaded core for tiny federates? Also, might a single multi-threaded core manage lots of federates and thereby take advantage of the threading effort (might need to be at the broker level)?
I am looking into a little what it would take to maintain both modes and not blow up the code maintenance, not sure yet.
My expectation is if you have multiple federates in the same process. the current system is faster. In separate process my rough guess would be when the # of federates exceed the number of available processor cores(SMP included) by more than 1.5x a single threaded helics core will probably be consistently faster.
Makes sense. Unless it is very easy to maintain, I would lean toward educating users to put lots of small federates in the same process through examples, tests, etc. Seems there are more important features to work out.
After poking around, I don't think creating a fully single fed core is going to be wise at present, mainly due to testing and maintenance of the messaging system and other parts of the code. What I am tinkering with is whether some things can be done easily in the existing architecture to reduce the threading if some options are chosen.
I have been poking at this issue for while, just exploring different options. And on the basis of some of the performance tests executed already. I think if you want to match/exceed some of the existing tests then it would be prudent to reduce the thread count. I think the best way of doing that is to divide up the configuration into two pieces. The first is a set of ST(single_thread) cores corresponding to the existing core types. The next is a flag in the federate specifying single federate. My best bet for minimizing code overhead would be to use the boost::Fiber library, with some tweaks. I think this could get the ST cores up reasonably quickly but based on the exploration I think it requires boost 1.66 or greater to work. So basically those core types would only be available if the library were compiled with boost 1.66 or greater. This would obviously be a longer term project but in some use cases it could be a significant performance gain. How do we feel about partitioning the capabilities of the core library based on the boost version.
This was partially addressed in the 3.3.0 release with the Callback federates. A second part will come with the single core federate planned for the 3.4 release.