Verbs memory pool may leak pinned memory when message is deleted on a PE different from the one to which it was delivered

[ericjbohm]

Original issue: https://charm.cs.illinois.edu/redmine/issues/1671

---

Receive a message on a node group Section multicast a pointer to that message to the pe's local to that host. Handle deletion by an atomic decrement refcount in the user message. Run for enough messages and you will exhaust the pinned memory, because the deleted message is not being handled correctly in the pool.

Probably replicable with a narrower test.

[stwhite91]

Original date: 2017-09-06 18:25:22

---

Note that this is specific to a mempool in src/arch/verbs/machine-ibverbs.c, not the generic mempool that we have for GNI and Isomalloc in src/arch/util/mempool.{c,h}

[stwhite91]

Original date: 2018-02-11 04:53:07

---

We don't really have a pressing need for this in 6.9.0, so deferring

[nitbhat]

Original date: 2019-03-15 14:59:20

---

I will try to reproduce this issue with a simple example.

[nitbhat]

Original date: 2019-04-03 15:22:59

---

Added a test case: https://charm.cs.illinois.edu/gerrit/c/charm/+/5055, but haven't been able to reproduce the bug so far.

[ericjbohm]

pinnedMemLeakReplicate branch of CharmVSS can be run on xyz_428 to replicate