openucx
How preallocate buffer through rendezvous protocol before ucp_tag_recv_nbx actually receiving?
It seams there is no api about rendezvous in UCP. And there is no clear document or example for UCT.
Hi, what do you mean "preallocate buffer through rendezvous protocol"? AFAIK you need to pass preallocated buffer to ucp_tag_recv_nbx as second argument.
To better understand how do UCP or UCT API works you can check examples
I would like to transmit the sparse tensors, and there is no way for two workers to know the size of the data in advance.
I don't know if I understand this correctly. In rendezvous protocol, ucp p2p sends the size of the data to be transmitted before the actual transmission begins. So when I use ucp, could I allocate a buffer based on the size of the rendezvous data? And then I pass this buffer into ucp_tag_recv_nbx.
Or is there a tag_recv API which can feed a allocator into.
I guess I need to refer to this to write my own API, right? https://github.com/openucx/ucx/blob/049eed4d7f5f0f8388054fcf068813427d6287e8/src/ucp/tag/tag_rndv.h
@MoFHeka for such case, it is recommended to use the active message API.
The sender can send its tensor using ucp_am_send_nbx, then a callback will be called on receiver with the length of the data and a "data" token. The callback can allocate a receive buffer and pass that token to ucp_am_recv_data_nbx, and complete the rendezvous data transfer. See https://github.com/openucx/ucx/blob/841b1b3/examples/ucp_client_server.c#L560 as example
@yosefe Which callback? ucp_am_recv_data_nbx_callback_t from ucp_request_param_t or ucp_am_recv_callback_t from ucp_am_handler_param_t. I'm a little confused about what these two callbacks are used for respectively.
- The first callback (ucp_am_recv_callback_t from ucp_am_handler_param_t) is for handling the initial message arrival and deciding how to handle the data
- The second callback (ucp_am_recv_data_nbx_callback_t from ucp_request_param_t) is for handling the completion of the actual data transfer operation This two-callback design allows for efficient handling of both eager and rendezvous protocols in UCX's Active Message API.
@yosefe Thank you for replying. I read the documentation and found that there is a ucp_am_callback_t that seems similar to the ucp_am_recv_data_nbx_callback_t. Did ucp_am_callback_t has being deprecated? Or it will be invoked at different time point compared with ucp_am_recv_data_nbx_callback_t. Or I am wrong, the ucp_am_callback_t is the same role as ucp_am_recv_callback_t?
Sorry for another question. What is purpose of the memh in ucp_request_param_t. It says is a pre-registered buffer. But what if I have already a pointer to ucp_am_recv_data_nbx, what does this memh buffer work for?
@yosefe Thank you for replying. I read the documentation and found that there is a ucp_am_callback_t that seems similar to the ucp_am_recv_data_nbx_callback_t. Did ucp_am_callback_t has being deprecated? Or it will be invoked at different time point compared with ucp_am_recv_data_nbx_callback_t. Or I am wrong, the ucp_am_callback_t is the same role as ucp_am_recv_callback_t?
- ucp_am_callback_t and the newer version ucp_am_recv_callback_t are used to process (handle) incoming active messages, they have the same role.
- ucp_am_recv_data_nbx_callback_t is the optional completion callback for ucp_am_recv_data_nbx() routine, that can be used to pull the receive data (after ucp_am_callback_t/ucp_am_recv_callback_t were called)
Sorry for another question. What is purpose of the memh in ucp_request_param_t. It says is a pre-registered buffer. But what if I have already a pointer to ucp_am_recv_data_nbx, what does this memh buffer work for?
It is an optional optimization parameter - for exampe if the user application manages memory pools, the memory pools can be registered with ucp_mem_map() and the returned memh can be passed to ucx along with the buffer from the corresponding memory pool. it reduces some overhead of ucx registration cache lookup.
Sorry for another question. What is purpose of the memh in ucp_request_param_t. It says is a pre-registered buffer. But what if I have already a pointer to ucp_am_recv_data_nbx, what does this memh buffer work for?
It is an optional optimization parameter - for exampe if the user application manages memory pools, the memory pools can be registered with ucp_mem_map() and the returned memh can be passed to ucx along with the buffer from the corresponding memory pool. it reduces some overhead of ucx registration cache lookup.
Hi @yosefe
I have a question for you. If iov data is sent, but iov_base may come from different memory regions, their memh is different, but ucp_request_param_t only supports specifying one memh, what should I do?
For iov data, the memh optimization parameter can be used only if all iov buffers are within the memh region.
For iov data, the memh optimization parameter can be used only if all iov buffers are within the memh region.
@yosefe Does this include am header buffer?
Sorry for another question. What is purpose of the memh in ucp_request_param_t. It says is a pre-registered buffer. But what if I have already a pointer to ucp_am_recv_data_nbx, what does this memh buffer work for?
It is an optional optimization parameter - for exampe if the user application manages memory pools, the memory pools can be registered with ucp_mem_map() and the returned memh can be passed to ucx along with the buffer from the corresponding memory pool. it reduces some overhead of ucx registration cache lookup.
@yosefe If I understand correctly, that means. In send_am, even if the buffer pointer and length are passed, if no corresponding memh map to buffer is passed, then ucp/uct will allocate a piece of memory and then copy the data corresponding to the buffer pointer?
@yosefe Does this include am header buffer?
The header buffer is not used in zero-copy mode so memh is not relevant for it
@yosefe If I understand correctly, that means. In send_am, even if the buffer pointer and length are passed, if no corresponding memh map to buffer is passed, then ucp/uct will allocate a piece of memory and then copy the data corresponding to the buffer pointer?
It depends on the message length. Small enough messages will be copied, larger messages will be auto-registered by UCX even if no explicit memh is passed, and sent in zero copy mode.