ompi OSC/UCX: Adding the following optimizations (nonblocking accumulate and reusing resources)

Adding the following optimizations: 1) Reuse the same workers/eps in single-threaded applications, this is helpful if an application creates many windows, therefore, we avoid the unnecessary overheads and 2) adding the truly nonblocking MPI_Accumulate/Get_Accumulate.

Signed-off-by: Mamzi Bayatpour [email protected] Co-authored-by: Tomislav Janjusic [email protected]

Aug 23 '22 23:08 MamziB

Can we split this into two PRs? The two changes seem to be independent.

Aug 24 '22 21:08 devreal

@devreal The features are independent, but the changes are intertwined, we'd prefer to leave it as is. FWIW, these are significant improvements in NWCHEM performance, now at least on small scale, we seem to outperform IMPI by about 20%, need to test larger scales to see effects.

Aug 24 '22 21:08 janjust

Hello! The Git Commit Checker CI bot found a few problems with this PR:

6b537d11: Fixing some corner cases in nonblocking accumulate...

check_signed_off: does not contain a valid Signed-off-by line
check_cherry_pick: contains a cherry pick message that refers to a commit that exists, but is in an as-yet unmerged pull request: ae97b9792e246f4397fc175fde36f41e7c35637f

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

Oct 06 '22 20:10 github-actions[bot]

@MamziB missing signed off in the last commit

Oct 06 '22 20:10 janjust

@MamziB missing signed off in the last commit

@janjust I am planning to squash all the new commits, and then add sign-off. Does that sound good to you?

Oct 06 '22 20:10 MamziB

Hello! The Git Commit Checker CI bot found a few problems with this PR:

addc6b82: enhance the datatype handling in nb acc

check_signed_off: does not contain a valid Signed-off-by line

6b537d11: Fixing some corner cases in nonblocking accumulate...

check_signed_off: does not contain a valid Signed-off-by line
check_cherry_pick: contains a cherry pick message that refers to a commit that exists, but is in an as-yet unmerged pull request: ae97b9792e246f4397fc175fde36f41e7c35637f

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

Oct 07 '22 18:10 github-actions[bot]

@devreal, Thanks for your constructive comments. Can you please let us know if you have more comments?

Oct 07 '22 21:10 MamziB

Hello! The Git Commit Checker CI bot found a few problems with this PR:

4afa6c7b: code reorganization

check_signed_off: does not contain a valid Signed-off-by line

addc6b82: enhance the datatype handling in nb acc

check_signed_off: does not contain a valid Signed-off-by line

6b537d11: Fixing some corner cases in nonblocking accumulate...

check_signed_off: does not contain a valid Signed-off-by line
check_cherry_pick: contains a cherry pick message that refers to a commit that exists, but is in an as-yet unmerged pull request: ae97b9792e246f4397fc175fde36f41e7c35637f

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

Oct 07 '22 21:10 github-actions[bot]

Hello! The Git Commit Checker CI bot found a few problems with this PR:

6db6d31d: Enhancing the osc finalize for resource utilizati...

check_signed_off: does not contain a valid Signed-off-by line

4afa6c7b: code reorganization

check_signed_off: does not contain a valid Signed-off-by line

addc6b82: enhance the datatype handling in nb acc

check_signed_off: does not contain a valid Signed-off-by line

6b537d11: Fixing some corner cases in nonblocking accumulate...

check_signed_off: does not contain a valid Signed-off-by line
check_cherry_pick: contains a cherry pick message that refers to a commit that exists, but is in an as-yet unmerged pull request: ae97b9792e246f4397fc175fde36f41e7c35637f

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

Oct 13 '22 18:10 github-actions[bot]

Hello! The Git Commit Checker CI bot found a few problems with this PR:

e4072fe7: fixing the macros

check_signed_off: does not contain a valid Signed-off-by line

5c28f90d: adding the prefix

check_signed_off: does not contain a valid Signed-off-by line

6db6d31d: Enhancing the osc finalize for resource utilizati...

check_signed_off: does not contain a valid Signed-off-by line

4afa6c7b: code reorganization

check_signed_off: does not contain a valid Signed-off-by line

addc6b82: enhance the datatype handling in nb acc

check_signed_off: does not contain a valid Signed-off-by line

6b537d11: Fixing some corner cases in nonblocking accumulate...

check_signed_off: does not contain a valid Signed-off-by line
check_cherry_pick: contains a cherry pick message that refers to a commit that exists, but is in an as-yet unmerged pull request: ae97b9792e246f4397fc175fde36f41e7c35637f

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

Oct 13 '22 20:10 github-actions[bot]

hey @devreal please let us know your further comments. Again, thanks for your various help so far.

Oct 13 '22 20:10 MamziB

Hello! The Git Commit Checker CI bot found a few problems with this PR:

08a847cc: use opal_uses_thread to set the mpi_thread_multipl...

check_signed_off: does not contain a valid Signed-off-by line

e4072fe7: fixing the macros

check_signed_off: does not contain a valid Signed-off-by line

5c28f90d: adding the prefix

check_signed_off: does not contain a valid Signed-off-by line

6db6d31d: Enhancing the osc finalize for resource utilizati...

check_signed_off: does not contain a valid Signed-off-by line

4afa6c7b: code reorganization

check_signed_off: does not contain a valid Signed-off-by line

addc6b82: enhance the datatype handling in nb acc

check_signed_off: does not contain a valid Signed-off-by line

6b537d11: Fixing some corner cases in nonblocking accumulate...

check_signed_off: does not contain a valid Signed-off-by line
check_cherry_pick: contains a cherry pick message that refers to a commit that exists, but is in an as-yet unmerged pull request: ae97b9792e246f4397fc175fde36f41e7c35637f

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

Oct 14 '22 17:10 github-actions[bot]

Hello! The Git Commit Checker CI bot found a few problems with this PR:

d91d76ae: atomic add for nb outstanding ops and renaming the...

check_signed_off: does not contain a valid Signed-off-by line

37163ce7: move num_incomplete_req_ops to osc ucx context

check_signed_off: does not contain a valid Signed-off-by line

08a847cc: use opal_uses_thread to set the mpi_thread_multipl...

check_signed_off: does not contain a valid Signed-off-by line

e4072fe7: fixing the macros

check_signed_off: does not contain a valid Signed-off-by line

5c28f90d: adding the prefix

check_signed_off: does not contain a valid Signed-off-by line

6db6d31d: Enhancing the osc finalize for resource utilizati...

check_signed_off: does not contain a valid Signed-off-by line

4afa6c7b: code reorganization

check_signed_off: does not contain a valid Signed-off-by line

addc6b82: enhance the datatype handling in nb acc

check_signed_off: does not contain a valid Signed-off-by line

6b537d11: Fixing some corner cases in nonblocking accumulate...

check_signed_off: does not contain a valid Signed-off-by line
check_cherry_pick: contains a cherry pick message that refers to a commit that exists, but is in an as-yet unmerged pull request: ae97b9792e246f4397fc175fde36f41e7c35637f

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

Oct 14 '22 23:10 github-actions[bot]

Hello! The Git Commit Checker CI bot found a few problems with this PR:

8213d67f: make nonblocking acc default

check_signed_off: does not contain a valid Signed-off-by line

d91d76ae: atomic add for nb outstanding ops and renaming the...

check_signed_off: does not contain a valid Signed-off-by line

37163ce7: move num_incomplete_req_ops to osc ucx context

check_signed_off: does not contain a valid Signed-off-by line

08a847cc: use opal_uses_thread to set the mpi_thread_multipl...

check_signed_off: does not contain a valid Signed-off-by line

e4072fe7: fixing the macros

check_signed_off: does not contain a valid Signed-off-by line

5c28f90d: adding the prefix

check_signed_off: does not contain a valid Signed-off-by line

6db6d31d: Enhancing the osc finalize for resource utilizati...

check_signed_off: does not contain a valid Signed-off-by line

4afa6c7b: code reorganization

check_signed_off: does not contain a valid Signed-off-by line

addc6b82: enhance the datatype handling in nb acc

check_signed_off: does not contain a valid Signed-off-by line

6b537d11: Fixing some corner cases in nonblocking accumulate...

check_signed_off: does not contain a valid Signed-off-by line
check_cherry_pick: contains a cherry pick message that refers to a commit that exists, but is in an as-yet unmerged pull request: ae97b9792e246f4397fc175fde36f41e7c35637f

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

Oct 17 '22 18:10 github-actions[bot]

Hello! The Git Commit Checker CI bot found a few problems with this PR:

ddf5d70b: cleanup

check_signed_off: does not contain a valid Signed-off-by line

dcb44cb0: cleanup

check_signed_off: does not contain a valid Signed-off-by line

8213d67f: make nonblocking acc default

check_signed_off: does not contain a valid Signed-off-by line

d91d76ae: atomic add for nb outstanding ops and renaming the...

check_signed_off: does not contain a valid Signed-off-by line

37163ce7: move num_incomplete_req_ops to osc ucx context

check_signed_off: does not contain a valid Signed-off-by line

08a847cc: use opal_uses_thread to set the mpi_thread_multipl...

check_signed_off: does not contain a valid Signed-off-by line

e4072fe7: fixing the macros

check_signed_off: does not contain a valid Signed-off-by line

5c28f90d: adding the prefix

check_signed_off: does not contain a valid Signed-off-by line

6db6d31d: Enhancing the osc finalize for resource utilizati...

check_signed_off: does not contain a valid Signed-off-by line

4afa6c7b: code reorganization

check_signed_off: does not contain a valid Signed-off-by line

addc6b82: enhance the datatype handling in nb acc

check_signed_off: does not contain a valid Signed-off-by line

6b537d11: Fixing some corner cases in nonblocking accumulate...

check_signed_off: does not contain a valid Signed-off-by line
check_cherry_pick: contains a cherry pick message that refers to a commit that exists, but is in an as-yet unmerged pull request: ae97b9792e246f4397fc175fde36f41e7c35637f

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

Oct 17 '22 19:10 github-actions[bot]

Hello! The Git Commit Checker CI bot found a few problems with this PR:

9f46f484: create separate req obj for accumulate

check_signed_off: does not contain a valid Signed-off-by line

ddf5d70b: cleanup

check_signed_off: does not contain a valid Signed-off-by line

dcb44cb0: cleanup

check_signed_off: does not contain a valid Signed-off-by line

8213d67f: make nonblocking acc default

check_signed_off: does not contain a valid Signed-off-by line

d91d76ae: atomic add for nb outstanding ops and renaming the...

check_signed_off: does not contain a valid Signed-off-by line

37163ce7: move num_incomplete_req_ops to osc ucx context

check_signed_off: does not contain a valid Signed-off-by line

08a847cc: use opal_uses_thread to set the mpi_thread_multipl...

check_signed_off: does not contain a valid Signed-off-by line

e4072fe7: fixing the macros

check_signed_off: does not contain a valid Signed-off-by line

5c28f90d: adding the prefix

check_signed_off: does not contain a valid Signed-off-by line

6db6d31d: Enhancing the osc finalize for resource utilizati...

check_signed_off: does not contain a valid Signed-off-by line

4afa6c7b: code reorganization

check_signed_off: does not contain a valid Signed-off-by line

addc6b82: enhance the datatype handling in nb acc

check_signed_off: does not contain a valid Signed-off-by line

6b537d11: Fixing some corner cases in nonblocking accumulate...

check_signed_off: does not contain a valid Signed-off-by line
check_cherry_pick: contains a cherry pick message that refers to a commit that exists, but is in an as-yet unmerged pull request: ae97b9792e246f4397fc175fde36f41e7c35637f

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

Oct 18 '22 17:10 github-actions[bot]

Hello! The Git Commit Checker CI bot found a few problems with this PR:

63e6893d: add description for mca vars

check_signed_off: does not contain a valid Signed-off-by line

9f46f484: create separate req obj for accumulate

check_signed_off: does not contain a valid Signed-off-by line

ddf5d70b: cleanup

check_signed_off: does not contain a valid Signed-off-by line

dcb44cb0: cleanup

check_signed_off: does not contain a valid Signed-off-by line

8213d67f: make nonblocking acc default

check_signed_off: does not contain a valid Signed-off-by line

d91d76ae: atomic add for nb outstanding ops and renaming the...

check_signed_off: does not contain a valid Signed-off-by line

37163ce7: move num_incomplete_req_ops to osc ucx context

check_signed_off: does not contain a valid Signed-off-by line

08a847cc: use opal_uses_thread to set the mpi_thread_multipl...

check_signed_off: does not contain a valid Signed-off-by line

e4072fe7: fixing the macros

check_signed_off: does not contain a valid Signed-off-by line

5c28f90d: adding the prefix

check_signed_off: does not contain a valid Signed-off-by line

6db6d31d: Enhancing the osc finalize for resource utilizati...

check_signed_off: does not contain a valid Signed-off-by line

4afa6c7b: code reorganization

check_signed_off: does not contain a valid Signed-off-by line

addc6b82: enhance the datatype handling in nb acc

check_signed_off: does not contain a valid Signed-off-by line

6b537d11: Fixing some corner cases in nonblocking accumulate...

check_signed_off: does not contain a valid Signed-off-by line
check_cherry_pick: contains a cherry pick message that refers to a commit that exists, but is in an as-yet unmerged pull request: ae97b9792e246f4397fc175fde36f41e7c35637f

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

Oct 18 '22 18:10 github-actions[bot]

Hello! The Git Commit Checker CI bot found a few problems with this PR:

18ff47b9: fix the naming of req comp

check_signed_off: does not contain a valid Signed-off-by line

63e6893d: add description for mca vars

check_signed_off: does not contain a valid Signed-off-by line

9f46f484: create separate req obj for accumulate

check_signed_off: does not contain a valid Signed-off-by line

ddf5d70b: cleanup

check_signed_off: does not contain a valid Signed-off-by line

dcb44cb0: cleanup

check_signed_off: does not contain a valid Signed-off-by line

8213d67f: make nonblocking acc default

check_signed_off: does not contain a valid Signed-off-by line

d91d76ae: atomic add for nb outstanding ops and renaming the...

check_signed_off: does not contain a valid Signed-off-by line

37163ce7: move num_incomplete_req_ops to osc ucx context

check_signed_off: does not contain a valid Signed-off-by line

08a847cc: use opal_uses_thread to set the mpi_thread_multipl...

check_signed_off: does not contain a valid Signed-off-by line

e4072fe7: fixing the macros

check_signed_off: does not contain a valid Signed-off-by line

5c28f90d: adding the prefix

check_signed_off: does not contain a valid Signed-off-by line

6db6d31d: Enhancing the osc finalize for resource utilizati...

check_signed_off: does not contain a valid Signed-off-by line

4afa6c7b: code reorganization

check_signed_off: does not contain a valid Signed-off-by line

addc6b82: enhance the datatype handling in nb acc

check_signed_off: does not contain a valid Signed-off-by line

6b537d11: Fixing some corner cases in nonblocking accumulate...

check_signed_off: does not contain a valid Signed-off-by line
check_cherry_pick: contains a cherry pick message that refers to a commit that exists, but is in an as-yet unmerged pull request: ae97b9792e246f4397fc175fde36f41e7c35637f

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

Oct 18 '22 18:10 github-actions[bot]

Hello! The Git Commit Checker CI bot found a few problems with this PR:

7651aa3b: fix thread_enabled name

check_signed_off: does not contain a valid Signed-off-by line

18ff47b9: fix the naming of req comp

check_signed_off: does not contain a valid Signed-off-by line

63e6893d: add description for mca vars

check_signed_off: does not contain a valid Signed-off-by line

9f46f484: create separate req obj for accumulate

check_signed_off: does not contain a valid Signed-off-by line

ddf5d70b: cleanup

check_signed_off: does not contain a valid Signed-off-by line

dcb44cb0: cleanup

check_signed_off: does not contain a valid Signed-off-by line

8213d67f: make nonblocking acc default

check_signed_off: does not contain a valid Signed-off-by line

d91d76ae: atomic add for nb outstanding ops and renaming the...

check_signed_off: does not contain a valid Signed-off-by line

37163ce7: move num_incomplete_req_ops to osc ucx context

check_signed_off: does not contain a valid Signed-off-by line

08a847cc: use opal_uses_thread to set the mpi_thread_multipl...

check_signed_off: does not contain a valid Signed-off-by line

e4072fe7: fixing the macros

check_signed_off: does not contain a valid Signed-off-by line

5c28f90d: adding the prefix

check_signed_off: does not contain a valid Signed-off-by line

6db6d31d: Enhancing the osc finalize for resource utilizati...

check_signed_off: does not contain a valid Signed-off-by line

4afa6c7b: code reorganization

check_signed_off: does not contain a valid Signed-off-by line

addc6b82: enhance the datatype handling in nb acc

check_signed_off: does not contain a valid Signed-off-by line

6b537d11: Fixing some corner cases in nonblocking accumulate...

check_signed_off: does not contain a valid Signed-off-by line
check_cherry_pick: contains a cherry pick message that refers to a commit that exists, but is in an as-yet unmerged pull request: ae97b9792e246f4397fc175fde36f41e7c35637f

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

Oct 18 '22 18:10 github-actions[bot]

Hello! The Git Commit Checker CI bot found a few problems with this PR:

5444d2e8: disable nb acc for dyn windows

check_signed_off: does not contain a valid Signed-off-by line

7651aa3b: fix thread_enabled name

check_signed_off: does not contain a valid Signed-off-by line

18ff47b9: fix the naming of req comp

check_signed_off: does not contain a valid Signed-off-by line

63e6893d: add description for mca vars

check_signed_off: does not contain a valid Signed-off-by line

9f46f484: create separate req obj for accumulate

check_signed_off: does not contain a valid Signed-off-by line

ddf5d70b: cleanup

check_signed_off: does not contain a valid Signed-off-by line

dcb44cb0: cleanup

check_signed_off: does not contain a valid Signed-off-by line

8213d67f: make nonblocking acc default

check_signed_off: does not contain a valid Signed-off-by line

d91d76ae: atomic add for nb outstanding ops and renaming the...

check_signed_off: does not contain a valid Signed-off-by line

37163ce7: move num_incomplete_req_ops to osc ucx context

check_signed_off: does not contain a valid Signed-off-by line

08a847cc: use opal_uses_thread to set the mpi_thread_multipl...

check_signed_off: does not contain a valid Signed-off-by line

e4072fe7: fixing the macros

check_signed_off: does not contain a valid Signed-off-by line

5c28f90d: adding the prefix

check_signed_off: does not contain a valid Signed-off-by line

6db6d31d: Enhancing the osc finalize for resource utilizati...

check_signed_off: does not contain a valid Signed-off-by line

4afa6c7b: code reorganization

check_signed_off: does not contain a valid Signed-off-by line

addc6b82: enhance the datatype handling in nb acc

check_signed_off: does not contain a valid Signed-off-by line

6b537d11: Fixing some corner cases in nonblocking accumulate...

check_signed_off: does not contain a valid Signed-off-by line
check_cherry_pick: contains a cherry pick message that refers to a commit that exists, but is in an as-yet unmerged pull request: ae97b9792e246f4397fc175fde36f41e7c35637f

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

Oct 19 '22 19:10 github-actions[bot]

Hello! The Git Commit Checker CI bot found a few problems with this PR:

174b53c3: adding ref count for accumulate lock

check_signed_off: does not contain a valid Signed-off-by line

5444d2e8: disable nb acc for dyn windows

check_signed_off: does not contain a valid Signed-off-by line

7651aa3b: fix thread_enabled name

check_signed_off: does not contain a valid Signed-off-by line

18ff47b9: fix the naming of req comp

check_signed_off: does not contain a valid Signed-off-by line

63e6893d: add description for mca vars

check_signed_off: does not contain a valid Signed-off-by line

9f46f484: create separate req obj for accumulate

check_signed_off: does not contain a valid Signed-off-by line

ddf5d70b: cleanup

check_signed_off: does not contain a valid Signed-off-by line

dcb44cb0: cleanup

check_signed_off: does not contain a valid Signed-off-by line

8213d67f: make nonblocking acc default

check_signed_off: does not contain a valid Signed-off-by line

d91d76ae: atomic add for nb outstanding ops and renaming the...

check_signed_off: does not contain a valid Signed-off-by line

37163ce7: move num_incomplete_req_ops to osc ucx context

check_signed_off: does not contain a valid Signed-off-by line

08a847cc: use opal_uses_thread to set the mpi_thread_multipl...

check_signed_off: does not contain a valid Signed-off-by line

e4072fe7: fixing the macros

check_signed_off: does not contain a valid Signed-off-by line

5c28f90d: adding the prefix

check_signed_off: does not contain a valid Signed-off-by line

6db6d31d: Enhancing the osc finalize for resource utilizati...

check_signed_off: does not contain a valid Signed-off-by line

4afa6c7b: code reorganization

check_signed_off: does not contain a valid Signed-off-by line

addc6b82: enhance the datatype handling in nb acc

check_signed_off: does not contain a valid Signed-off-by line

6b537d11: Fixing some corner cases in nonblocking accumulate...

check_signed_off: does not contain a valid Signed-off-by line
check_cherry_pick: contains a cherry pick message that refers to a commit that exists, but is in an as-yet unmerged pull request: ae97b9792e246f4397fc175fde36f41e7c35637f

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

Oct 20 '22 19:10 github-actions[bot]

@devreal Please kindly let us know if you have more comments. Thanks a lot.

Oct 20 '22 19:10 MamziB

Hello! The Git Commit Checker CI bot found a few problems with this PR:

00ebe716: Enhance ref counters

check_signed_off: does not contain a valid Signed-off-by line

174b53c3: adding ref count for accumulate lock

check_signed_off: does not contain a valid Signed-off-by line

5444d2e8: disable nb acc for dyn windows

check_signed_off: does not contain a valid Signed-off-by line

7651aa3b: fix thread_enabled name

check_signed_off: does not contain a valid Signed-off-by line

18ff47b9: fix the naming of req comp

check_signed_off: does not contain a valid Signed-off-by line

63e6893d: add description for mca vars

check_signed_off: does not contain a valid Signed-off-by line

9f46f484: create separate req obj for accumulate

check_signed_off: does not contain a valid Signed-off-by line

ddf5d70b: cleanup

check_signed_off: does not contain a valid Signed-off-by line

dcb44cb0: cleanup

check_signed_off: does not contain a valid Signed-off-by line

8213d67f: make nonblocking acc default

check_signed_off: does not contain a valid Signed-off-by line

d91d76ae: atomic add for nb outstanding ops and renaming the...

check_signed_off: does not contain a valid Signed-off-by line

37163ce7: move num_incomplete_req_ops to osc ucx context

check_signed_off: does not contain a valid Signed-off-by line

08a847cc: use opal_uses_thread to set the mpi_thread_multipl...

check_signed_off: does not contain a valid Signed-off-by line

e4072fe7: fixing the macros

check_signed_off: does not contain a valid Signed-off-by line

5c28f90d: adding the prefix

check_signed_off: does not contain a valid Signed-off-by line

6db6d31d: Enhancing the osc finalize for resource utilizati...

check_signed_off: does not contain a valid Signed-off-by line

4afa6c7b: code reorganization

check_signed_off: does not contain a valid Signed-off-by line

addc6b82: enhance the datatype handling in nb acc

check_signed_off: does not contain a valid Signed-off-by line

6b537d11: Fixing some corner cases in nonblocking accumulate...

check_signed_off: does not contain a valid Signed-off-by line
check_cherry_pick: contains a cherry pick message that refers to a commit that exists, but is in an as-yet unmerged pull request: ae97b9792e246f4397fc175fde36f41e7c35637f

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

Oct 21 '22 22:10 github-actions[bot]

@devreal I'm going to push the last commit. Please wait for that. I am working on pushing it.

Oct 24 '22 23:10 MamziB

Hello! The Git Commit Checker CI bot found a few problems with this PR:

a2d2993b: Using a separate lock for handling the dynamic win...

check_signed_off: does not contain a valid Signed-off-by line

00ebe716: Enhance ref counters

check_signed_off: does not contain a valid Signed-off-by line

174b53c3: adding ref count for accumulate lock

check_signed_off: does not contain a valid Signed-off-by line

5444d2e8: disable nb acc for dyn windows

check_signed_off: does not contain a valid Signed-off-by line

7651aa3b: fix thread_enabled name

check_signed_off: does not contain a valid Signed-off-by line

18ff47b9: fix the naming of req comp

check_signed_off: does not contain a valid Signed-off-by line

63e6893d: add description for mca vars

check_signed_off: does not contain a valid Signed-off-by line

9f46f484: create separate req obj for accumulate

check_signed_off: does not contain a valid Signed-off-by line

ddf5d70b: cleanup

check_signed_off: does not contain a valid Signed-off-by line

dcb44cb0: cleanup

check_signed_off: does not contain a valid Signed-off-by line

8213d67f: make nonblocking acc default

check_signed_off: does not contain a valid Signed-off-by line

d91d76ae: atomic add for nb outstanding ops and renaming the...

check_signed_off: does not contain a valid Signed-off-by line

37163ce7: move num_incomplete_req_ops to osc ucx context

check_signed_off: does not contain a valid Signed-off-by line

08a847cc: use opal_uses_thread to set the mpi_thread_multipl...

check_signed_off: does not contain a valid Signed-off-by line

e4072fe7: fixing the macros

check_signed_off: does not contain a valid Signed-off-by line

5c28f90d: adding the prefix

check_signed_off: does not contain a valid Signed-off-by line

6db6d31d: Enhancing the osc finalize for resource utilizati...

check_signed_off: does not contain a valid Signed-off-by line

4afa6c7b: code reorganization

check_signed_off: does not contain a valid Signed-off-by line

addc6b82: enhance the datatype handling in nb acc

check_signed_off: does not contain a valid Signed-off-by line

6b537d11: Fixing some corner cases in nonblocking accumulate...

check_signed_off: does not contain a valid Signed-off-by line
check_cherry_pick: contains a cherry pick message that refers to a commit that exists, but is in an as-yet unmerged pull request: ae97b9792e246f4397fc175fde36f41e7c35637f

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

Oct 24 '22 23:10 github-actions[bot]

@devreal Please take a look at the last commits. I reverted a couple of commits, as they were not MPI atomicity compliant. Basically, using those two commits, one process can initiate multiple on-the-flight nonblocking accumulate to the same target. This can violate the atomicity of each MPI_Accumulate/Get_accumulate call. Now everything should be good to go.

Oct 25 '22 00:10 MamziB

I squashed the commits, however, since my branch is not based on the latest main branch, some unwanted commits sneaked in. I will rebase my branch on top of the latest main and I will try again. Thanks for your patience.

Oct 26 '22 18:10 MamziB

bot:retest

Oct 31 '22 13:10 janjust

bot:retest

Oct 31 '22 19:10 janjust

bot:ibm:retest

Nov 01 '22 21:11 janjust

ompi ompi copied to clipboard

OSC/UCX: Adding the following optimizations (nonblocking accumulate and reusing resources)

ompi
ompi copied to clipboard