Add functions for packet allocate, free

[jasmeetbagga]

Sending a packet requires allocating it first. Typically, this requires allocating from a special pool of DMAed memory. For this a ASIC vendor SDK may support custom allocate and free functions. So mirror those in the SAI spec.

Without such a ability we are left to allocate heap memory,
pass it to send_hostif_packet_fn, which then allocates from the
afore mentioned DMA memory, copies the contents of heap memory
and sends the packet along. This causes us to have a extra
allocation and copy. This can induce a substantial slowdown -
~30% in our measurements

[jasmeetbagga]

https://github.com/jasmeetbagga/SAI/commit/72257b22e2b966f254be4b6ff4ce6fcff6b3ba9b

[kcudnik]

this is valid point, but that way SAI would need to expose some alloc/free functions to actually accomplish that, usually copy memory is fast, and as you suggest in your setup extra copy takes 30% of time, then seems like there is nothing else going on with that packet ? just to put it to dma? do you have any numbers like pkts/sec or bytes/sec how fast this is with/without copy

i think this is good topic to bring up on SAI community meeting i also saw you commit, and i dont understand why alloc/free has also attributes lists on them, maybe there could be some special attributes on alloc, to tell dma channel or other stuff, but why free would need attributes ?

[jasmeetbagga]

Thanks for reviewing

do you have any numbers like pkts/sec or bytes/sec how fast this is with/without copy

I have numbers internally, but probably can't talk about them here w/o violating the ASIC vendor NDA. The way I am comparing this is that for FBOSS - https://github.com/facebook/fboss, we are in the process of converting from native SDK code to SAI. The code using native SDK was following the allocate DMA, write packet there and send out approach vs with SAI we had to do an allocate in user space, copy to DMA, send packet out and then free both DMA and user space memory. I wrote a simple benchmark of sending packets as fast as possible from CPU and measure bps, pps. The SAI approach gives 30% poorer performance.

also saw you commit, and i dont understand why alloc/free has also attributes lists on them, maybe there could be some special attributes on alloc, to tell dma channel or other stuff, but why free would need attributes ?

You are right about alloc, depending upon the tx type (https://github.com/opencomputeproject/SAI/blob/master/inc/saihostif.h#L1101) packet allocation may set different flags on the allocated memory. This is not a must for my use case, as this flag setting can happen on packet send as well. On the free side, I retained a symmetric interface in case some implementations had different deallocation behavior based on the attributes (say deallocated from one DMA pool vs another). Having said that, my use case doesn't need attributes on either call, so I am open to feedback on this.

[kcudnik]

free DMA memory one poll vs another should not take place, since you pass exact buffer pointer to the free function and base on that pointer deallocation should happen internally but i dont know internal details how that dma with pools is implemented, just basing on common alloc/free headers used in industry
also there is another way to recv/send packets using SAI_HOSTIF_TYPE_NETDEV, but this will require some kernel code, but it would expose eth/Etherenet interfaces on linux which adds easy access via socket on any application

[jasmeetbagga]

I removed attr parameters from free - 35335d9d30247c6782e1b89728fadfefb080fe4a

[kcudnik]

hostif packet send/recv functions have also attr_list parameter which is attribute list, and for utilize this, we introduced artificial object type SAI_OBJECT_TYPE_HOSTIF_PACKET which corresponds to enum sai_hostif_packet_attr_t which contains attributes that are used by send/rect functions, and not mixed with sai_hostif_attr_t attributes. from your allocate function: https://github.com/jasmeetbagga/SAI/commit/72257b22e2b966f254be4b6ff4ce6fcff6b3ba9b#diff-88814cd3ce988236bffc44e4085ea0f9a450ac20d3eee7a859d22a2eb13ec76cR1280 seems like new attributes that corresponds to allocate method, should be added to sai_hostif_packet_attr_t enum, is that right ?

[jasmeetbagga]

@kcudnik that's correct. As discussed in the meeting today added a attribute there to convey zero copy TX - d6e5820a95d104f04e855aa39a98715ad136fdfe

[kcudnik]

ok, LGTM, if you don't have any other changes i think you can create PR for that