Multithreading support clarification

[vspetrov]

We declare 3 types of threading support but we don't explicitly say which APIs are allowed to be used concurrently from multiple threads:

• obviously user can use collectives_init/post/test/finalize with multiple threads (with the proper thread level support)
• Can user call ucc_context_create from multiple threads ? Using different ucc_lib_h handles?
• Can user call ucc_team_create_post/test/destroy? I guess if the context is the same then not. But what about different contexts?

[manjugv]

From the spec:

"Invocation semantics: The ucc_team_create_post is a nonblocking collective operation, in which the partic- ipants are determined by the user-provided OOB collective operation. Overlapping of multiple ucc_team_←􏰀 create_post operations are invalid. Posting a collective operation before the team handle is created is invalid. The team handle is destroyed by a blocking collective operation; the participants of this collective operation are the same as the create operation. When the user does not provide an OOB collective operation, all participants calling the ucc_create_post operation will be part of a new team created."

[manjugv]

Does this address some of the questions?

[vspetrov]

@manjugv this does not really answer all the questions above. This does specifies that team_create_post/test/destroy can only be called from 1 thread. But what about ucc_context_create?

Secondly: i don't understand this statement: When the user does not provide an OOB collective operation, all participants calling the ucc_create_post operation will be part of a new team created. - does this mean we allow team-create w/o oob ? Does this mean user has to pass smth else, e.g. list of EPs or smth like that ? Where is it explained in the spec?

[manjugv]

1. For context creation, we don’t want to place this restriction. Because you want to support the use case where you want to create resource per thread and that is exclusively used by that thread.

2. Yes, this gives flexibility for the user not to implement an OOB collective. In this case, the library will be responsible for the OOB. It will make the assumption that all processes that will call ucc_init will be part of the team.

[vspetrov]

@manjugv

1. I think we still should prohibit concurrent context creations from multiple threads. The user should create contexts and then give them to different threads this is fine, but not creating them concurrently. The reason: we have a mode when the OOB is passed to ctx create, in this case the concurrent calls to OOB will result in the undefined ordering and likely a hang.
2. How will ucc implement anything internally w/o having means of communication and any info about the "world". Do you imply PMIx integration there ?

[vspetrov]

Additional MT questions (@Sergei-Lebedev following our discussion today): assuming user initialized ucc ctx with THREAD_MULTIPLE,

• Is it allowed to call ucc_collective_finalize(request_1) and ucc_collective_init/post(request_2) concurrently from 2 different threads on the SAME team ? Ie, concurrent calls to collective_init/post and finalize
• Is it allowed to call ucc_context_progress(ctx) in 1 thread and ucc_team_create/destroy from the same process in another thread?

[manjugv]

1. I think we still should prohibit concurrent context creations from multiple threads. The user should create contexts and then give them to different threads this is fine, but not creating them concurrently. The reason: we have a mode when the OOB is passed to ctx create, in this case the concurrent calls to OOB will result in the undefined ordering and likely a hang.

In this case, the responsibility is on the user to ensure that no two OOB calls are concurrent, correct? Given that local context creation is a common case, I’m not sure we need to place such restrictions. This is the problem with making context create both local and collective operation (we debated this). :) If we really need such semantic, I think it will be clean, if we keep two interfaces for context creation: one with collective semantics, and the other with local semantics.

[manjugv]

2. How will ucc implement anything internally w/o having means of communication and any info about the "world". Do you imply PMIx integration there ?

Yes.

[manjugv]

• Is it allowed to call ucc_collective_finalize(request_1) and ucc_collective_init/post(request_2) concurrently from 2 different threads on the SAME team ? Ie, concurrent calls to collective_init/post and finalize

Since init and finalize operations are collective calls, it violates the thread model semantics. So, I would say no.

[manjugv]

• Is it allowed to call ucc_context_progress(ctx) in 1 thread and ucc_team_create/destroy from the same process in another thread?

Since context is a local entity, I would yes. To truly avoid such scenarios, we need to maintain one context per thread. Remember, currently, we don’t have a thread model for context. Rather, the thread model is for the whole library. We decided against defining thread model for context.