UCT/API: Introduce shared mkey buffer

[dmitrygx]

What

Introduce shared mkey buffer in UCT.

Why ?

Shared mkey buffer will be packed by uct_md_mkey_pack_v2.

How ?

1. Introduce UCT_MD_FLAG_SHARED_MKEY MD capability flag.
2. Update uct_md_mkey_pack_v2 to support packing shared mkey.
3. Introduce shared_mkey_packed_size and global_id MD attributes.
4. Update all transports to set shared_mkey_packed_size and global_id.

[dmitrygx]

@yosefe @shamisp @brminich @tonycurtis could you review pls? btw, we should decided on the name of the mkey, we have the following options:

• Shared
• Crossed
• Foreign
• Alias
• Virtual

[yosefe]

@yosefe @shamisp @brminich @tonycurtis could you review pls? btw, we should decided on the name of the mkey, we have the following options:

• Shared
• Crossed
• Foreign
• Alias
• Virtual

Maybe "external"?

[dmitrygx]

@yosefe @shamisp @brminich @tonycurtis could you review pls? btw, we should decided on the name of the mkey, we have the following options:

• Shared
• Crossed
• Foreign
• Alias
• Virtual

Maybe "external"?

it seems that all options could be split into two groups:

• suitable for exporter (HOST) and importer (DPU): shared, virtual, alias
• suitable for either exporter (HOST) or importer (DPU): crossed - suitable only for DPU (crossing - on HOST), foreign - suitable only for DPU, external - suitable only for DPU

imo, "alias" could mean:

• on HOST: a mkey could be aliased
• on DPU: a mkey is alias of some mkey created on HOST.

[yosefe]

maybe "inter" or "ipc" ?

[brminich]

maybe "inter" or "ipc" ?

we already have rkey (stands for remote), which is similar to inter. ipc is better in this sense

[dmitrygx]

maybe "inter" or "ipc" ?

we already have rkey (stands for remote), which is similar to inter. ipc is better in this sense

"IPC mkey" sounds great because even though processes are on different CPUs, this key could be used by a process to access memory registered by a process-exporter

[dmitrygx]

maybe "inter" or "ipc" ?

we already have rkey (stands for remote), which is similar to inter. ipc is better in this sense

"IPC mkey" sounds great because even though processes are on different CPUs, this key could be used by a process to access memory registered by a process-exporter

@shamisp @tonycurtis what do you think about "IPC" name? Just imagine that "IPC" is used instead of "SHARED" in all places of this PR.

[tonycurtis]

On Aug 16, 2022, at 5:53 PM, dmitrygx @.***> wrote:

maybe "inter" or "ipc" ?

we already have rkey (stands for remote), which is similar to inter. ipc is better in this sense

"IPC mkey" sounds great because even though processes are on different CPUs, this key could be used by a process to access memory registered by a process-exporter

@shamisp https://github.com/shamisp @tonycurtis https://github.com/tonycurtis what do you think about "IPC" name? Just imagine that "IPC" is used instead of "SHARED" in all places of this PR.

Sounds ok to me, Tony

[shamisp]

Based on discussion with @dmitrygx , he should follow up with Jason to make sure that our naming aligns with potential rdma-core naming. The concern that was raised about IPC naming is that actual behavior of the key does not align with IPC.

[dmitrygx]

Based on discussion with @dmitrygx , he should follow up with Jason to make sure that our naming aligns with potential rdma-core naming. The concern that was raised about IPC naming is that actual behavior of the key does not align with IPC.

@shamisp @yosefe @brminich @tonycurtis we discussed Cross-GVMI feature with @jgunthorpe, rdma-core won't introduce it in Verbs. so, we are free to select any name for our API and make it usable by other SmartNICs, if they will have the same functionality.

[shamisp]

Based on discussion with @dmitrygx , he should follow up with Jason to make sure that our naming aligns with potential rdma-core naming. The concern that was raised about IPC naming is that actual behavior of the key does not align with IPC.

@shamisp @yosefe @brminich @tonycurtis we discussed Cross-GVMI feature with @jgunthorpe, rdma-core won't introduce it in Verbs. so, we are free to select any name for our API and make it usable by other SmartNICs, if they will have the same functionality.

I think my final vote will go to "exported". Shared is fine but it can be confused with shared memory especially in the context of UCX, which has shared memory transports.

[dmitrygx]

Based on discussion with @dmitrygx , he should follow up with Jason to make sure that our naming aligns with potential rdma-core naming. The concern that was raised about IPC naming is that actual behavior of the key does not align with IPC.

@shamisp @yosefe @brminich @tonycurtis we discussed Cross-GVMI feature with @jgunthorpe, rdma-core won't introduce it in Verbs. so, we are free to select any name for our API and make it usable by other SmartNICs, if they will have the same functionality.

I think my final vote will go to "exported". Shared is fine but it can be confused with shared memory especially in the context of UCX, which has shared memory transports.

ok, I think "exported" should be fine, because in UCP we have the following flow:

• HOST: we create an exported mkey using ucp_mem_map(EXPORTED, params, &mkey)
• HOST: we serialize the exported mkey using ucp_memh_pack(mkey, &exported_mkey_buffer)
• DPU: create ucp_mem_map(EXPORTED, params = { exported_mkey_buffer }, &mkey) - I think EXPORTED should be ok here and we don't need to add IMPORTEDflag.

@yosefe @brminich @tonycurtis

[yosefe]

ok, I think "exported" should be fine, because in UCP we have the following flow:

👍

[dmitrygx]

@yosefe could you review pls?

[dmitrygx]

@yosefe squashed, could you pls approve?

[dmitrygx]

@tonycurtis @shamisp could you review pls?

[dmitrygx]

@tonycurtis your comments have been applied. @brminich @shamisp @yosefe @tonycurtis could you review pls? no need to wait for CI completion for the next review round, since only comments were fixed.

[yosefe]

@dmitrygx i think u didnt push

[dmitrygx]

@dmitrygx i think u didn't push

yes, forgot...

[dmitrygx]

@shamisp @brminich could you take a look pls?