crossbeam-rs
Feature: Ability to register callback on receiver/sender message queue transitions
After much testing and futzing about with select, I found that I had to abandon it as it didn't scale well into the 1000s of channels. Instead, I've had to resort to wrapping the channel and intercepting Send calls in order to deal with avoiding blocks and scheduling recv. What I'd perf is to have a way to register a callback when the message queue, for a receiver, transitions from empty to non-empty and for a sender, transitions from full to non-full. That would allow me to no longer need to wrap. I'd then be able to remove the code I have that schedules a receiver task when sending. I'd also be able to drop the polling code that sends when the channel is full. You can see my unfortunate implementation of this in d3-core. I haven't look at how difficult it might be to implement, but I'd be willing to take on that task if you'd like to assign it to me.
After much testing and futzing about with select, I found that I had to abandon it as it didn't scale well into the 1000s of channels.
Do you have a reproducer where I can see this behavior? I am quite curious how it really scales/performs.
I do. You can fork my d3 repo here and checkout the 0.1.1 tag. In there you'll find a test-server example. You can configure various exercisers. I usually create 5000 machines in a daisy chain. Anyway, I run a single thread to run the select and 3 threads running executors with a deque between them. I squeezed as much as a I could out of select, but between the shuffle per call and having to walk a large portion of the list to find a ready element, it just didn't scale.
In the end I wrapped the crossbeam sender and receiver and use send to schedule the receiver (if it isn't already scheduled). That works pretty well, I even manage to block on full without blocking the executor thread. One of the uglies I do have is managing drop of the receiver (as a result of the last sender dropping) as that has to turn into a notification. Ideally, I'd want to schedule each time the queue goes from empty to non-empty and each time it goes from full to non-full, so that I can schedule a receiver and sender. Senders, where there are multiple of them into a single receiver are an interesting little problem all their own. While order can't matter, you can end up with a thundering herd problem. Where the queue is full and 1-2 slots open up and 50 senders are waiting and well, you know what happens. Now, imagine it with a bunch of threads executing...ya, most are doing the meaningless task of find that it is still full. I've got a nice solution worked out, I just haven't had the time to implement it and I'd prefer to implement it on a layer that provides queue transition notifications.
I've recently created a github to compare and contrast various channel implementation -- both async and sync. There I simulate a server, running a 5 element pipeline (a real server might have more, but it pretty much matches what an audio bridge might have), and vary the number of lanes from a few hundred to a few thousand (representing connections). Anyway, you can see that select is not your friend when you get past a few hundred, ever with multiple threads. Feel free to have a look at https://github.com/BruceBrown/channel-executor-experiment I'm updating it today to include more crates. If you see something I did wrong with crossbeam, or could have done better feel free to file a ticket or even better a pull request. Thanks,