xdp-tutorial
xdp-tutorial copied to clipboard
xdp-project
advanced03-AF_XDP: XDP_SHARED_UMEM
How would the code be changed to make use of XDP_SHARED_UMEM ? Is it possible to add an extra section for this? And/or to add an extra tutorial assignment?
Sure; contributions welcome! :)
Or maybe try to bug @chaudron, @magnus-karlsson or @bjoto to add this?
I asked @magnus-karlsson (who added XDP_SHARED_UMEM to the xdpsock example code in the Linux source tree) via email and I'm putting his response here in case it helps anybody else:
[Simon] I'm trying to find a working example of using XDP_SHARED_UMEM and came across your commit here "Add XDP_SHARED_UMEM support to xdpsock" [1]. Interestingly, neither the example code nor the commit code seem to use the XDP_SHARED_UMEM flag, which is only mentioned in this string [2] for the xdpsock command line usage.
[Magnus] libbpf sets this flag automatically if you register the same umem more than twice, so no need to use it in you application if you use libbpf, which I recommend. So the xdpsock sample uses it indirectly.
[1] https://github.com/torvalds/linux/commit/2e5d72c15f0dc713c203464c5c76eb4ec285f598#diff-adae9c74ffdf99d8001c2686f8e21faf [2] https://github.com/torvalds/linux/blob/master/samples/bpf/xdpsock_user.c#L655
On Mon, Mar 2, 2020 at 8:21 PM Simon Hardy-Francis [email protected] wrote:
I asked @magnus-karlsson https://github.com/magnus-karlsson (who added XDP_SHARED_UMEM to the xdpsock example code in the Linux source tree) via email and I'm putting his response here in case it helps anybody else:
[Simon] I'm trying to find a working example of using XDP_SHARED_UMEM and came across your commit here "Add XDP_SHARED_UMEM support to xdpsock" [1]. Interestingly, neither the example code nor the commit code seem to use the XDP_SHARED_UMEM flag, which is only mentioned in this string [2] for the xdpsock command line usage.
[Magnus] libbpf sets this flag automatically if you register the same umem more than twice, so no need to use it in you application if you use libbpf, which I recommend. So the xdpsock sample uses it indirectly.
I should of course read "the same umem more than once" not "more than twice". Sorry ;-).
/Magnus
[1] torvalds/linux@2e5d72c#diff-adae9c74ffdf99d8001c2686f8e21faf https://github.com/torvalds/linux/commit/2e5d72c15f0dc713c203464c5c76eb4ec285f598#diff-adae9c74ffdf99d8001c2686f8e21faf [2] https://github.com/torvalds/linux/blob/master/samples/bpf/xdpsock_user.c#L655
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/xdp-project/xdp-tutorial/issues/107?email_source=notifications&email_token=AASGUENQ5MQRU3ORJIYGF7TRFQBNNA5CNFSM4K7ACZU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENQTD6A#issuecomment-593572344, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASGUEPLQLIJJVMHANUPDVDRFQBNNANCNFSM4K7ACZUQ .
[@magnus-karlsson ] libbpf sets this flag automatically if you register the same umem more than twice, so no need to use it in you application if you use libbpf, which I recommend.
So in theory to change the tutorial advanced03-AP_XDP code to support XDP_SHARED_UMEM from two separate processes on the same box, I'd need to allocate both UMEM and its wrapper structure as shared memory? Currently neither the wrapper structure allocated via configure_xsk_umem() [1] nor the UMEM allocated here [2] are allocated using shared memory.
Assuming they are both allocated as the same shared memory then the libbpf function xsk_socket__create() will automatically use the XDP_SHARED_UMEM flag due to the refcount member of the UMEM wrapper structure here [3] ?
Is there a recommended way to allocate both the UMEM wrapper structure and UMEM itself as shared memory? I was thinking of creating two files in /dev/shm and memory map those RAM backed memory mapped files for the UMEM wrapper structure and UMEM itself. Would that work, or is there a better way?
[1] https://github.com/xdp-project/xdp-tutorial/blob/master/advanced03-AF_XDP/af_xdp_user.c#L126 [2] https://github.com/xdp-project/xdp-tutorial/blob/master/advanced03-AF_XDP/af_xdp_user.c#L565 [3] https://github.com/libbpf/libbpf/blob/master/src/xsk.c#L701
On Tue, Mar 3, 2020 at 6:51 PM Simon Hardy-Francis [email protected] wrote:
[@magnus-karlsson https://github.com/magnus-karlsson ] libbpf sets this flag automatically if you register the same umem more than twice, so no need to use it in you application if you use libbpf, which I recommend.
So in theory to change the tutorial advanced03-AP_XDP code to support XDP_SHARED_UMEM from two separate processes on the same box, I'd need to allocate both UMEM and its wrapper structure as shared memory? Currently neither the wrapper structure allocated via configure_xsk_umem() [1] nor the UMEM allocated here [2] are allocated using shared memory.
Assuming they are both allocated as the same shared memory then the libbpf function xsk_socket__create() will automatically use the XDP_SHARED_UMEM flag due to the refcount member of the UMEM wrapper structure here [3] ?
Is there a recommended way to allocate both the UMEM wrapper structure and UMEM itself as shared memory? I was thinking of creating two files in /dev/shm and memory map those RAM backed memory mapped files for the UMEM wrapper structure and UMEM itself. Would that work, or is there a better way?
How about using pthreads?
/Magnus
[1] https://github.com/xdp-project/xdp-tutorial/blob/master/advanced03-AF_XDP/af_xdp_user.c#L126 [2] https://github.com/xdp-project/xdp-tutorial/blob/master/advanced03-AF_XDP/af_xdp_user.c#L565 [3] https://github.com/libbpf/libbpf/blob/master/src/xsk.c#L701
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/xdp-project/xdp-tutorial/issues/107?email_source=notifications&email_token=AASGUEI6COH7NGIRADQ3JYTRFU7RHA5CNFSM4K7ACZU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENUPKJI#issuecomment-594081061, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASGUEL6NXNUHOSPPNCJDD3RFU7RHANCNFSM4K7ACZUQ .
[@magnus-karlsson ] libbpf sets this flag automatically if you register the same umem more than twice, so no need to use it in you application if you use libbpf, which I recommend.
So in theory to change the tutorial advanced03-AP_XDP code to support
XDP_SHARED_UMEMfrom two separate processes on the same box, I'd need to allocate both UMEM and its wrapper structure as shared memory? Currently neither the wrapper structure allocated viaconfigure_xsk_umem()[1] nor the UMEM allocated here [2] are allocated using shared memory.Assuming they are both allocated as the same shared memory then the libbpf function
xsk_socket__create()will automatically use theXDP_SHARED_UMEMflag due to therefcountmember of the UMEM wrapper structure here [3] ?Is there a recommended way to allocate both the UMEM wrapper structure and UMEM itself as shared memory? I was thinking of creating two files in
/dev/shmand memory map those RAM backed memory mapped files for the UMEM wrapper structure and UMEM itself. Would that work, or is there a better way?[1] https://github.com/xdp-project/xdp-tutorial/blob/master/advanced03-AF_XDP/af_xdp_user.c#L126 [2] https://github.com/xdp-project/xdp-tutorial/blob/master/advanced03-AF_XDP/af_xdp_user.c#L565 [3] https://github.com/libbpf/libbpf/blob/master/src/xsk.c#L701
Hi @simonhf ,
Have you ever successfully create the /dev/shm for afxdp ?
Or do you come up with any better way?
On Mon, Dec 12, 2022 at 6:17 AM junka @.***> wrote:
@.*** https://github.com/magnus-karlsson ] libbpf sets this flag automatically if you register the same umem more than twice, so no need to use it in you application if you use libbpf, which I recommend.
So in theory to change the tutorial advanced03-AP_XDP code to support XDP_SHARED_UMEM from two separate processes on the same box, I'd need to allocate both UMEM and its wrapper structure as shared memory? Currently neither the wrapper structure allocated via configure_xsk_umem() [1] nor the UMEM allocated here [2] are allocated using shared memory.
Assuming they are both allocated as the same shared memory then the libbpf function xsk_socket__create() will automatically use the XDP_SHARED_UMEM flag due to the refcount member of the UMEM wrapper structure here [3] ?
Is there a recommended way to allocate both the UMEM wrapper structure and UMEM itself as shared memory? I was thinking of creating two files in /dev/shm and memory map those RAM backed memory mapped files for the UMEM wrapper structure and UMEM itself. Would that work, or is there a better way?
[1] https://github.com/xdp-project/xdp-tutorial/blob/master/advanced03-AF_XDP/af_xdp_user.c#L126 [2] https://github.com/xdp-project/xdp-tutorial/blob/master/advanced03-AF_XDP/af_xdp_user.c#L565 [3] https://github.com/libbpf/libbpf/blob/master/src/xsk.c#L701
libxdp and libbpf (before AF_XDP support was deprecated in version 1.0) assumes shared memory for the umem wrapper struct. If you do not have this, you need to communicate the umem information in some way between the processes and then change some information such as the base address of the umem, as it likely will be different between the processes. Someone did try this out and succeeded, but never shared the code publically, unfortunately.
Hi @simonhf https://github.com/simonhf ,
Have you ever successfully create the /dev/shm for afxdp ? Or do you come up with any better way?
— Reply to this email directly, view it on GitHub https://github.com/xdp-project/xdp-tutorial/issues/107#issuecomment-1345896053, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASGUELI6T2VOR5AATM525TWM2YMZANCNFSM4K7ACZUQ . You are receiving this because you were mentioned.Message ID: @.***>
@magnus-karlsson Thx for you info
Quote from https://www.kernel.org/doc/html/networking/af_xdp.html
The UMEM can be shared between processes, if desired. If a process wants to do this, it simply skips the registration of the UMEM and its corresponding two rings, sets the XDP_SHARED_UMEM flag in the bind call and submits the XSK of the process it would like to share UMEM with as well as its own newly created XSK socket. The new process will then receive frame addr references in its own RX ring that point to this shared UMEM. Note that since the ring structures are single-consumer / single-producer (for performance reasons), the new process has to create its own socket with associated RX and TX rings, since it cannot share this with the other process. This is also the reason that there is only one set of FILL and COMPLETION rings per UMEM. It is the responsibility of a single process to handle the UMEM.
From doc above, it seems we don't have to share umem address. Only rx/tx rings should be created in a second process. Is that correct ?
On Mon, Dec 12, 2022 at 9:11 AM junka @.***> wrote:
@magnus-karlsson https://github.com/magnus-karlsson Thx for you info
Quote from https://www.kernel.org/doc/html/networking/af_xdp.html
The UMEM can be shared between processes, if desired. If a process wants to do this, it simply skips the registration of the UMEM and its corresponding two rings, sets the XDP_SHARED_UMEM flag in the bind call and submits the XSK of the process it would like to share UMEM with as well as its own newly created XSK socket. The new process will then receive frame addr references in its own RX ring that point to this shared UMEM. Note that since the ring structures are single-consumer / single-producer (for performance reasons), the new process has to create its own socket with associated RX and TX rings, since it cannot share this with the other process. This is also the reason that there is only one set of FILL and COMPLETION rings per UMEM. It is the responsibility of a single process to handle the UMEM.
From doc above, it seems we don't have to shae umem address. Only rx/tx rings should be created in a second process. Is that correct ?
I do not know what you mean by "share umem address". Both processes have to be able to reach the same umem area, otherwise it would not be a "shared umem" by definition. Libxdp and libbpf assume they share the same address space, but the kernel code and the raw uapi do not. Note that the case above is if you bind two sockets to the same netdev and queue_id. You can share a umem between different netdevs and queue_ids too, but then it is a different process described later in the document.
— Reply to this email directly, view it on GitHub https://github.com/xdp-project/xdp-tutorial/issues/107#issuecomment-1346054611, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASGUEIIQTXJL67UQZZI6H3WM3M2VANCNFSM4K7ACZUQ . You are receiving this because you were mentioned.Message ID: @.***>
I do not know what you mean by "share umem address".
I mean when two process access a memory address, they need to know the address out of the process namespace.
So either they have to use /dev/shm or hugepages.
But from kernel documentation, the second process does not have to know or access the umem address pointer. We may just transmit the umem->fd to the second process for bind.
Both processes have to be able to reach the same umem area, otherwise it would not be a "shared umem" by definition. Libxdp and libbpf assume they share the same address space, but the kernel code and the raw uapi do not. Note that the case above is if you bind two sockets to the same netdev and queue_id. You can share a umem between different netdevs and queue_ids too, but then it is a different process described later in the document.
If I understand correctly. While umem, cq and fq are used in kernel and the first process. And only one set of cq and fq supported per umem. The second process does not need that. So with libxdp/libbpf or not, we do not have to know or setup the umem for the second process. (expect the fd for bind)
On Mon, Dec 12, 2022 at 9:55 AM junka @.***> wrote:
I do not know what you mean by "share umem address".
I mean when two process access a memory address, they need to know the address out of the process namespace. So either they have to use /dev/shm or hugepages.
That is not a requirement per say. They only have to know the base address of the umem in their own address space and of course share the same umem area by some means.
But from kernel documentation, the second process does not have to know or access the umem address pointer. We may just transmit the umem->fd to the second process for bind.
If the second process does not access the umem, what is the point of sharing it then? Maybe I am not understanding your use case.
Both processes have to be able to reach the same umem area, otherwise it would not be a "shared umem" by definition. Libxdp and libbpf assume they share the same address space, but the kernel code and the raw uapi do not. Note that the case above is if you bind two sockets to the same netdev and queue_id. You can share a umem between different netdevs and queue_ids too, but then it is a different process described later in the document.
If I understand correctly. While umem, cq and fq are used in kernel and the first process. And only one set of cq and fq supported per umem. The second process does not need that. So with libxdp/libbpf or not, we do not have to know or setup the umem for the second process. (expect the fd for bind)
That is correct for the case when both sockets share the same netdev and queue id. But I still think the second process would like to access the umem :-).
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>
Have you ever successfully create the
/dev/shmfor afxdp ? Or do you come up with any better way?
Hi @junka, I did get this all to work back in early 2020 even without the examples being updated :-) But I no longer have access to the source code or my notes, and my memory is foggy regarding nitty gritty details. But I definitely had this working back then: Multiple processes accessing THE SAME packets via the same shared memory structures backed (note: it's not really backed; there's only one copy of the RAM, i.e. each page exists only once) by /dev/shm. Without this working then the only solution would be for a single process to read the packets (creating a potential bottleneck) and then somehow share the read packets (annoying and/or expensive IPC overhead) with all the interested processes via IPC somehow. HTH.