ch4: use am_tag_{send,recv} in RMA get/put

[hzhou]

Pull Request Description

Use am_tag_{send,recv} in RMA get/put.

This potentially works around the issue #7118

[skip warnings]

TODO

• [x] Get
• [x] Put

Author Checklist

• [x] Provide Description Particularly focus on why, not what. Reference background, issues, test failures, xfail entries, etc.
• [x] Commits Follow Good Practice Commits are self-contained and do not do two things at once. Commit message is of the form: module: short description Commit message explains what's in the commit.
• [ ] Passes All Tests Whitespace checker. Warnings test. Additional tests via comments.
• [x] Contribution Agreement For non-Argonne authors, check contribution agreement. If necessary, request an explicit comment from your companies PR approval manager.

[hzhou]

test:mpich/ch4/ucx test:,mpich/ch4/ofi/more

[hzhou]

test:mpich/ch4/most test:mpich/ch4/ofi/more

[abrooks98]

@hzhou can you rebase this on main and resolve conflicts? Now that some GPU fixes are in to resolve memory issues, we are hoping this will now fix #7118

[hzhou]

test:mpich/ch4/most test:mpich/ch4/ofi/more

[hzhou]

@hzhou can you rebase this on main and resolve conflicts? Now that some GPU fixes are in to resolve memory issues, we are hoping this will now fix #7118

Sure. Please test on aurora. Last the the user reported still seeing the hang with this PR.

[raffenet]

Is it worth updating this PR to use the new rndv/pipeline infrastructure? Anyway worth retesting as-is with https://github.com/pmodels/mpich/issues/7118. Will update the PR to facilitate testing.

[hzhou]

Updated.