From Poll to Interrupt driven...

[cycl0ne]

Hi, i have a question regarding this cool stack. I noticed, that all communication is poll based (calling pico_tick every x ms). Now I wonder, if it is intentional? Reading wiki about drivers says: IRQ based, but the code is still polling drivers even when IRQing.

I also found in the Wiki : pico_signal_wait()

Something which could be an idea of an event based system, not polling every x ms. But nowhere in Source used. Is it for a future plan? Or is it implemented somewhere in the other branches?

At the moment im trying to integrate something similar into the stack and first changed the timer handling (extracting from pico_stack. into an own file + handling of a sorted delta list): https://github.com/cycl0ne/picotcp/blob/dev-timer/stack/pico_timer.c Now one could spawn the timer_check() into a timerthread (of your fav. OS).

I wouldnt like to do the job of "unpolling" code, if this is already done or in work.

Cheers C.

[danielinux]

Hello @cycl0ne

picoTCP has experimental support as interrupt-driven stack, in a branch called 'tickless. It looks like a possibility that the branch will be considered for further research by the Board in the future. This has been discussed in the last picoTCP community meeting. You should check the official status with @frederikvs .

Meanwhile, seems like the branch is a bit stalling lately, so we forked here to add some improvements/fixes and integrate with frosted OS. In particular, you should look at this commit .

The concept is the following: pico_stack_tick is replaced by pico_stack_go which instead of polling all the layers, only executes pending actions. Since picoTCP manages timers internally, the pico_stack_go function returns the number of milliseconds to the next timed-event, which is the maximum time interval that can elapse before the function itself needs to be called again. In case of an interrupt, or an API socket event, the function needs to be executed again. pico_stack_go returns 0 if there are still actions pending, meaning "call again", and it returns a negative value when there are no timers pending, meaning that you can safely wait for the next external event before calling again.

You can figure out of how this is implemented by looking at frosted tcp/ip kernel code in low-power mode .

In any case, a device driver can omit the implementation of the poll method, use a deferred routine of an interrupt handler to execute one of the receiving functions upon frame arrivals (e.g. pico_stack_recv or pico_stack_recv_zerocopy_ext_buffer_notify) asynchronously. The tickless stack will add actions to process the incoming frames, so the OS should resume the pico_stack_go execution as soon as a frame arrives.

A similar approach is implemented in frosted usb_ethernet driver when running in LOW_POWER mode.

[frederikvs]

thanks @danielinux for the extensive answer :-)

Currently there are no plans with the tickless branch yet. It is an interesting direction to take, but we simply lack the time to work on it.

@cycl0ne if you feel like working on it, feel free to do so. One thing that I want to be upfront about : if you want your modifications to be merged into the main branch, we'd ask you to sign our CLA. This enables us to use picoTCP under other licenses as well, which keeps our corporate sponsors happy - and it's these corporate sponsors who have enabled us to build this stack, and to keep working on it. I hope you understand :-)

Thanks for your interest!

[cycl0ne]

Hi, thanks for the explanation. I already dug deep into the code an noticed that the _GO approach is not so fine in my eyes, since you still work synchronous through your stack, putting frames from one queue into the next, calling each module in sync (in -> network -> ethernet -> ....ethernet -> out). A better approach would be to split up into worker tasks maybe (Arp-Daemon, ICMP-Daemon,.. ), but i'm not sure in the moment to achieve this "platform" independent. I could imagine something like this: driver <- worker task read() -> ethernet -> if (icmp) put frame queue then signal ICMPD ....ICMPD send() -> ethernet -> write() -> driver it would be a complete different approach and im still in the "thinking" phase on how one could match your API with a Tickless (eventbased) API doing it "asyncstyle".

At the moment i feel ok on my own branch, if you later like my approach, i can still sign this and you can merge this into main.

[danielinux]

Thanks for sharing your view. I still prefer to go for a monolithic single-threaded TCP/IP stack in the kernel, because I have the impression that moving frame pointers between queues is more efficient than having to coordinate switching (kernel) threads, especially if this require context changes. Workers are being scheduled in order, in the same context and without delays if you implement the loop properly, which means that with the _go approach it would look like:

• eth interrupt for RX, dsr is schedule via jobs interface, WFI() is interrupted
• when interrupt returns, TCP/IP jobs are checked, DSR is executed, which calls pico_stack_recv
• the stack recv will schedule ethernet_recv
• ethernet_recv will schedule ipv4_recv
• ipv4_recv will schedule icmp4_recv
• ... the ping is bounced back through the stack by executing exactly the needed operations in the right sequence
• before the control is returned to the kernel loop, the ping reply is already in the TX ethernet queue, no copy involved, nothing else happens in supervisor mode (unless other IRQ are triggered in the meanwhile).

If anyway you want to explore multi-threaded picoTCP, you should implement pico_mutex_* functions to ensure that the access to shared resources is exclusive (e.g. queues). You can find an example in the POSIX implementation (see modules/posix.c).

What I really like about your approach is the possibility to have the timer events handled externally from the stack. At the moment frosted implements the same binary heap structure twice, but TCP timers and kernel timers are separated. I'll probably look more to this approach to improve the integration with frosted.

[cycl0ne]

Hmm, depends on the platform you support. Context switch yes, but supervisor mode? On most of the embedded OS you shouldnt have a supervisor there, since you mostly also have no mmu/vmm. So 99% of all code should run in userland. And i didnt want to put all into threads. Only things, which maybe need some triggers, but as i said, i still need some more understanding of the code. :)

Another thing i noticed it the approach driver calling pico_tcp. i am allways fond of drivers being standalone and the "program" which uses the driver needs to call the driver. Otherwise the driver is too much "married" with the implementation. What you think about it, switching the call pico_stack_recv() to a read_driver() ?

And this is where i would see a thread there, where each driver has an own.

pseudocode:

static void pico_driver_thread(drivername)
{
dev = opendevmod(drivername);
while (true) 
{
    len = read_device(dev, buffer, size);
    pico_stack_recv(buffer, len);
}
}

void pico_startdevmod(drivername)
{
    spawnthread(pico_driver_thread, drivername);
}

with this you dont need to poll the device. if it has data, it will send it to the thread waking it up and then the thread can send it to the whole stack. One more poll less on the stack_tick().

[danielinux]

ARM support for supervisor mode has nothing to do with vmm/mmu. Frosted for example runs on cortex-M no-MMU in real multiprocess mode (Posix profile 54) with separate supervisor/user memory. We've chosen to run tickless picoTCP entirely in kernel context to minimize the latency of the packet management and to avoid locking as much as possible. Of course the posix threads require a copy of the payloads to/from userspace, but that's the price to pay for the posix compliance.

If we are talking about integrating TCP/IP in kernel, I still think that single thread is the best approach. If you in fact plan to run the stack in userland, you can indeed still have the stack run in one or more threads to distribute the load. We took a similar approach while porting to Mbed or FreeRTOS, but indeed as previously mentioned you need to be careful with locking and mutexes especially across queues.

If you plan to have portable drivers (i.e. running on different stacks), then it's more difficult to have a pure synchronous approach. The last solution you proposed heavily depends on the thread scheduling (assuming that the read_device() is blocking on a semaphore), and might not be ideal in terms of performance, unless you fine-tune your thread priorities and deadlines thoroughly.

Again, I am not a big fan of the threads approach, especially when time-sharing with TCP/IP. But we can safely agree to disagree here :smile:

Anyway I am enjoying very much this conversation, and I will be happy to follow your findings and know more about your design approach. I also want to thank the picoTCP community for hosting this interesting thread over here :+1:

[cycl0ne]

Hi, depends on your implementation. On my RaspberryPi implementation i only used the supervisor stack to switch tasks. the "kernel" runs in userspace. i just want to be flexible as possible on the implementation. and most of the "network drivers" out there are read()/write() driven, without the driver calling the network stack. making poting of these easier. i know that single threading is maybe the eaisiest approach, but more and more embedded systems are also getting more cores. the Pine64 board i have here is also using multi cpu. i just need to find some time next to my job (and the weather which getting better) to better get into the code.at the moment im lacking some spare time :(

yes the discussion is really nice. too bad we are here in the bug tracker ;-)

[frederikvs]

We've had talks before about opening another communication channel, but came to no real conclusion. @cycl0ne : what do you have in mind? Forum? Mail list? Something else?

And yes, most drivers out there are probably read()/write(), but converting that to our model requires just a short wrapper, doing the read() and then calling pico_stack_recv(). An advantage to the pico_stack_recv() is that we also have support for other things, like the pico_stack_recv_zerocopy(), which gives us better efficiency, and improved performance.