[HELP] CONFIG_IOB_THROTTLE does not do what it claims

[mennovf]

Description

I ran into an issue with a UDP socket where a single task sequentially reads from and writes to a blocking, no timeout UDP socket. CONFIG_NET_RECV_BUFSIZE was 0, so the only place where the UDP readahead IOB buffers get freed is in recvfrom. If the IOB free list got emptied before the sendto() call, that call would block indefinitely waiting for memory, never giving the recvfrom() a chance to produce IOB buffers.

From its description in kconfig, I got the impression that CONFIG_IOB_THROTTLE was specifically created to prevent this kind of deadlock (although it only speaks of TCP, it's also used in the UDP code). On inspecting the UDP stack, if CONFIG_IOB_THROTTLE is enabled, the UDP reception allocates with throttle=true as expected. However the send logic of both TCP and UDP also allocate throttled instead of unthrottled, contradicting the IOB_THROTTLE description and thus not solving my issue. udp_wrbuffer_alloc however DOES alloc unthrottled but appears to never get called.

I am not directly in control of the code which sequentially sendto()/recvfrom()s the socket to fix the behaviour by e.g. moving them to separate tasks. My question is whether this IOB_THROTTLE behaviour is a bug, or whether there's another intended (config) solution.

Verification

• [x] I have verified before submitting the report.

[acassis]

@mennovf yes, I think it is a BUG (or at least a contradictory behavior according to documentation).

Possible solutions:

• Use a positive value to CONFIG_NET_RECV_BUFSIZE this way the readhead buffers are freed early
• Other alternative is running recv and send in different threads, but as you said you don't have access to the control flow it is not an option
• Increase the total IOB pool size, it could "mask" the issue, that could be enough for some low-end (low network performance) IoT devices

The proper solution should fix the network code to use unthrottled buffer allocation to avoid the outbond sends been starved by read-ahead.

@jkivilin @linguini1 @xiaoxiang781216 @wengzhe any suggestion?

[linguini1]

I am not familiar enough with this area, sorry

[fdcavalcanti]

Weirdly I think this relates to #16915 and has been bugging me for eternity.

[mennovf]

• Use a positive value to CONFIG_NET_RECV_BUFSIZE this way the readhead buffers are freed early

Sadly this is not a solution as it is also bugged. The comparison is made against the FILLED size of the buffer, not the size of the whole buffer. In my test it wasn't difficult to get into the situation where the io_pktlen was much smaller than rcvbufs even though all buffers were allocated to the readahead queue, especially if packing is turned off.

[acassis]

• Use a positive value to CONFIG_NET_RECV_BUFSIZE this way the readhead buffers are freed early

Sadly this is not a solution as it is also bugged. The comparison is made against the FILLED size of the buffer, not the size of the whole buffer. In my test it wasn't difficult to get into the situation where the io_pktlen was much smaller than rcvbufs even though all buffers were allocated to the readahead queue, especially if packing is turned off.

Also @fdcavalcanti found another issue here:

https://github.com/apache/nuttx/pull/17358#issuecomment-3575721610