Support affinity/pinning of parallel flows to different CPUs

[marcosfsch]

Enhancement Request

• Current behavior iperf3 supports setting affinity to a single CPU, which made sense when it was single threaded

• Desired behavior Support setting a rage/list of CPUs to affinity of multiple flows: Ex: "iperf3 -c localhost -P 4 -A 1-4,1-4"

• Implementation notes There are at least two way this can be implementated and I'd suggest the second approach:

1. Replicate numactl behavior, where "numactl -C 1-4" would bind each flow to the CPU range "1-4" and Linux would be responsible for the scheduling, allowing for dinamically rebalancing the flows but generating noise and perfromance drops.
2. Statically schedule each flow to a different CPU, i.e. in a round-robin fashion, so that when you have a maching set of parallel threads and CPUs you'd have 1 flow per CPU, minimizing reschedules during the transfer and optimizing performance

Another note, is that ideally you should be able to explicitly define a CPU list, which normally uses comma as a separator, i.e. "1,3,5,7". But this would impact either in I) changing the current client/server CPU separator ("iperf3 -c localhost -P 4 -A 1,3,5,7/2,4,6,8") or to use a different delimiter character for defining the list.

[bmah888]

You raise a good point in that the current -A behavior doesn't work very well in a multi-threaded iperf3. So far our standard practice within ESnet is just to do numactl, as you suggested in the first approach. How bad are the downsides you mentioned...have you or others observed these problems? (I don't think we have, but multithreaded application performance analysis is not my forte.)

In considering these different implementations, we also want to keep in mind other OS platforms that support -A, such as FreeBSD (a supported platform), and Windows (while not officially supported, an environment that I'd like to avoid gratuitously breaking).

[davidBar-On]

Submitted PR #1778 with a suggested multi-CPUs Affinity support.

[manedurphy]

I happened to be exploring an idea related to the affinity feature and stumbled upon this existing request. What I was looking into was adding the ability to pin specific streams/threads to target CPU cores. If I run an iperf3 client on an 4-core machine, then I should be able to pin each stream/thread to each core of that machine: iperf3 --client my.iperf3-server.com --parallel 4 --stream-affinity 0,1,2,3. Implemented this with a comma-separated list in iperf/#1939, but it's just a proof-of-concept, not the final design. I raised the PR as reference material for expanding the conversation on this enhancement request.

@davidBar-On, if I'm understanding your implementation, the user could something like this: iperf3 --client my.iperf3-server.com --parallel 4 --affinity 0/2-4 which tells iperf3 that it is welcome to run on CPUs 0, 2, 3, 4, and that any of the streams/threads created from --parallel are allowed to run on any of those CPUs, but which CPUs they run on is decided by the Linux scheduler, right? My curiosity is whether we should give the user even more control by allowing them to pin each stream/thread as desired. I think there may be value to implementing both, where any of these would be valid:

• iperf3 --client my.iperf3-server.com --parallel 4 --affinity 0/2-4, limits the process to CPUs 0, 2, 3, 4
• iperf3 --client my.iperf3-server.com --parallel 3 --stream-affinity 0/2-4/1, assigns streams/threads to CPUs:
  • stream/thread 1 -> CPU0
  • stream/thread 2 -> CPU2,CPU3,CPU4
  • stream/thread 3 -> CPU1

What are your thoughts on this?

[davidBar-On]

@manedurphy I see that you already got a response to your PR and that it maybe that both yours and this PR will not be used.

Your proposal is interesting. However, since all iperf3 streams are the same, I am not sure what is the use case for assigning specific streams to specific CPUs. This is unless you have use case where each CPU performance is different, you want to test what happens if you assign 2 streams to one CPU, 3 streams to another, etc.

In any case, I think that if your proposal is merged, this or similar solution should also be merged, as it is more important to allow to set which CPUs should be used than assigning specific streams to specific CPUs.