Possible deadlock when using mesh with high contention on the queues

[geobeau]

When taking the sharding example: https://github.com/DataDog/glommio/blob/master/examples/sharding.rs If you increase the number of messages in the stream or reduce the channel size, the program will never finish.

Step to reproduce:

bash -c "cd examples; cargo run --example channel_mesh" # Example is working well, finishing properly

Modify the channel_size if the mesh: let mesh = MeshBuilder::full(nr_shards, 1024); To let mesh = MeshBuilder::full(nr_shards, 1);

bash -c "cd examples; cargo run --example channel_mesh" # Now it never finishes

You can also reproduce by increase the "take" size: let messages = repeat_with(|| fastrand::i32(0..100)).take(10).inspect(move |x| println!("shard {} generated {}", me, x)); To let messages = repeat_with(|| fastrand::i32(0..100)).take(1000000).inspect(move |x| println!("shard {} generated {}", me, x));

Environment:

• Rust version: rustc 1.61.0 (fe5b13d68 2022-05-18)
• Glommio version: Master (0cffeef51a89f477008f96bf1f1d9c5eaecee98d) and v0.7.0
• Linux version: Linux 5.10.0-0.bpo.12-amd64 #1 SMP Debian 5.10.103-1~bpo10+1 (2022-03-08) x86_64 GNU/Linux

[glommer]

Thanks! Realistically it will take me a couple of weeks to get to this as I am currently traveling. sorry about that!