sahlberg
question: libev integration
Hi,
As I understand from cursory browsing of libnfs everything happens in eventloop (rpc_service/rpc_timeout_scan()) which has to be executed in current thread. Is it possible to integrate this loop with libev loop? Or maybe adapt libnfs library to use libev loop?
Background: I am looking for a way to use libnfs in C/C++ application that uses coroutines (libev + libcoro, maybe libevfibers). I.e. most threads will have a personal libev event loop and most logic will be implemented as coroutines (I'll have to wrap libnfs async API for this to work).
Thank you, Michael.
Hi,
I would want to avoid integrating libev into libnfs, just because I want to limit the number external dependencies.
It should however not be too to integrate it into multithreaded application where each thread has its own event loop. The main issues you need to be careful about is that libnfs is not threads safe so you will need to protect all calls to *_service() by a mutex so that only one thread at a time can call into it.
You also need a different mutex for calling into any/all *_async() so that only a single thread at a time enters the normal libnfs functions.
Make sure to only use the _async() functions and avoid the sync versions of the calls as they would otherwise run their own internal poll based event loop.
I recently added multithread safety to fuse-nfs. You can look at the recent commits in that project for ideas on how to make sure that you call libnfs in a thread safe fashion. I think the fuse-nfs code should be close to what you want for running one event loop in each thread.
regards ronnie sahlberg
On Thu, Feb 8, 2018 at 2:23 PM, crusader-mike [email protected] wrote:
Hi,
As I understand from cursory browsing of libnfs everything happens in eventloop (rpc_service/rpc_timeout_scan()) which has to be executed in current thread. Is it possible to integrate this loop with libev loop? Or maybe adapt libnfs library to use libev loop?
Background: I am looking for a way to use libnfs in C/C++ application that uses coroutines (libev + libcoro, maybe libevfibers). I.e. most threads will have a personal libev event loop and most logic will be implemented as coroutines (I'll have to wrap libnfs async API for this to work).
Thank you, Michael.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/sahlberg/libnfs/issues/241, or mute the thread https://github.com/notifications/unsubscribe-auth/AAeNkBGmjneGI99AK4jZLQKUeI2ZAWgXks5tSna4gaJpZM4R9x_S .
No, integrating libev would be a bad idea. I was thinking about having tools to integrate libnfs event loop with external one. I.e. something similar to what libcurl is doing (see example for libuv integration)
The main issues you need to be careful about is that libnfs is not threads safe so you will need to protect all calls to *_service() by a mutex so that only one thread at a time can call into it. You also need a different mutex for calling into any/all *_async() so that only a single thread at a time enters the normal libnfs functions.
Wait, is it not threadsafe at all or threadsafe as long as rpc_service()/*_async() are called with different rpc/nfs context? I am fine with latter -- each thread will have it's own context.
I recently added multithread safety to fuse-nfs
I need to talk to NFS server directly (without POSIX API layer in-between) in my asynchronous (event-loop-based) application. NFSv3 is fine, NFSv4 is just a bonus at this stage. I spent some time looking around for NFS client library and seems that libnfs may work for me (maybe after few changes). Please, let me know if I am wrong.
On Fri, Feb 16, 2018 at 2:33 PM, crusader-mike [email protected] wrote:
No, integrating libev would be a bad idea. I was thinking about having tools to integrate libnfs event loop with external one. I.e. something similar to what libcurl is doing (see example for libuv integration https://raw.githubusercontent.com/curl/curl/master/docs/examples/multi-uv.c )
The main issues you need to be careful about is that libnfs is not threads safe so you will need to protect all calls to *_service() by a mutex so that only one thread at a time can call into it. You also need a different mutex for calling into any/all *_async() so that only a single thread at a time enters the normal libnfs functions.
Wait, is it not threadsafe at all or threadsafe as long as rpc_service() is called with different rpc/nfs context? I am fine with latter -- each thread will have it's own context.
If you have a different nfs/rpc context for each thread, then it is fully thread-safe. There are no global variables shared across the contexts so as long as each thread has its own context then you are fine.
My note was only if you want a single nfs/rpc context and share it across multiple threads. It is sometimes more convenient to just have a single context and share it across all threads. For that case you need to so something like I did in fuse-nfs. (since all _async() functions are non-blocking, it will not hurt performance eventhough this means you will have to serialize all nfs/rpc function calls via a futex/mutex)
I recently added multithread safety to fuse-nfs
I need to talk to NFS server directly (without POSIX API layer in-between) in my asynchronous (event-loop-based) application. NFSv3 is fine, NFSv4 is just a bonus at this stage. I spent some time looking around for NFS client library and seems that libnfs may work for me (maybe after few changes). Please, let me know if I am wrong.
I think libnfs should work for you and it is designed primarily to use the async interface. The sync interface is only meant for trivial applications where you either don't care about performance or it is just not worth it to use a proper event system like libev or libtevent.
I am sure that libnfs will work for you. If there are issues or you need changes or new feature, just let me know and I will be happy to add them.
Regards
Ronnie Sahlberg
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/sahlberg/libnfs/issues/241#issuecomment-366141746, or mute the thread https://github.com/notifications/unsubscribe-auth/AAeNkKKsOQ-_pyYJrZm_2SY0Z4CdZfHSks5tVQUQgaJpZM4R9x_S .
I think libnfs should work for you and it is designed primarily to use the async interface.
Great! This brings us to original question -- is there any way to integrate libnfs event loop into external event loop (e.g. libev/libuv)? :) ... maybe something similar to what libcurl did?
On Fri, Feb 16, 2018 at 3:04 PM, crusader-mike [email protected] wrote:
I think libnfs should work for you and it is designed primarily to use the async interface.
Great! This brings us to original question -- is there any way to integrate libnfs event loop into external event loop (e.g. libev/libuv)? :) ... maybe something similar to what libcurl did?
The only place libnfs uses an internal event loop is in libnfs-sync.c This is basically for the sync version of the high-level nfs/vfs based API and as a convenience for people that wants to write a trivial app and don't want to have to deal with setting up a proper event system. This is done using poll in wait_for_nfs_reply().
The async nfs interface or the low-level rpc based API do not have any event loop in libnfs at all. It is expected that those users will want to write their own event loop, using their system of preference, and thus for those use cases I provide :
[nfs|rpc]_get_fd() : so you know which FD to wait on in your event loop
[nfs|rpc]_which_events() : A combination of POLLIN and POLLOUT which is what you should tell the event library you want to trigger on
and
[nfs|rpc]_service() : Which is what you call from your event loop if/when the *_get_fd() descriptor becomes readable/writeable according to what *_which_events() returned.
You will probably need to write some trivial wrappers to interface get_fd/which_events/service with your event system. See QEMU block/iscsi.c for an example on how that interfaceing was done there.
regards ronnie sahlberg
—
You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/sahlberg/libnfs/issues/241#issuecomment-366144937, or mute the thread https://github.com/notifications/unsubscribe-auth/AAeNkAC7USjdqC53U2H-9yjTrO_tgOfGks5tVQxDgaJpZM4R9x_S .
Perfect! That is exactly what I need.
I've checked "send path" to understand how you deal with multiple outgoing requests. As I understood:
- if udp is used for transport -- request gets immediately sent out (via
sendto()) - if tcp -- request is added to
rpc->outqueueand (after user calls related*_service()) it gets pushed into the socket - in both cases hash map
rpc->waitpdugets a new entry that would allow "read path" to associate response with original request
Related questions:
- It seems that in udp case we end up using "blocking" sendto() -- which can block in some cases or drop packets (if outgoing buffer is full). Am I wrong? I never dealt with "non-blocking" UDP sockets... Are they better in this context?
- For tcp case it makes sense (if
outqueueis empty) to send data out immediately (and add remainder, if any, intooutqueue) -- this would improve latency. Would you agree? - in udp case memory for incoming response (see
rpc_read_from_socket()) can be allocated on stack to avoidmalloc()/free(). Is this a bad idea? - NFS server often drops tcp connections after some timeout. I've seen
rpc_set_autoreconnect()/etcin the source code, but just to make sure -- does libnfs auto-reconnect int this case? How long it'll keep trying? (e.g. if NFS server goes down) How long it waits between reconnect attempts?
On Wed, Feb 21, 2018 at 1:54 PM, crusader-mike [email protected] wrote:
Perfect! That is exactly what I need.
I checked "send path" to understand how you deal with multiple outgoing
requests. As I understand:
- if udp is used for transport -- request gets immediately sent out (via sendto())
- if tcp -- request is added to rpc->outqueue and (after user calls related *_service()) it gets pushed into the socket
- in both cases hash map rpc->waitpdu gets a new entry that would allow "read path" to associate response with original request
Yes. Note however that UDP never really used by the normal nfs/rpc functions. TCP is the default and is what should be used for everything.
The only exception is the way that libnfs uses to try to discover NFS servers on the local network. This is done using "broadcast rpc" and this can only be done using UDP. (this is the same thing as "rpcinfo -b 100005 3" uses.)
This is the only reason UDP support exists in libnfs. Don't use UDP.
Related questions:
- It seems that in udp case we end up using "blocking" sendto() https://stackoverflow.com/questions/39804326/non-blocking-sendto-function -- which can block in some cases or drop packets (if outgoing buffer is full). Am I wrong? I never dealt with "non-blocking" UDP sockets... Are they better in this context?
Just ignore the UDP case and only use TCP.
- For tcp case it makes sense (if outqueue is empty) to send data out immediately (and add remainder, if any, into outqueue) -- this would improve latency. Would you agree?
I don't think it should have much/any impact on performance unless you are completely singlethreaded. The simplest solution if you need to shave off this latency is just manually calling *_service(POLLOUT) immediately after you have called the _async() function. That way you dont have to wait for your event loop to trigger.
It is nice if the *_async functions do invoke _service() since this allows you to queue up several _async calls at once and then send them out in one single tcp segment.
- in udp case memory for incoming response (see rpc_read_from_socket()) can be allocated on stack to avoid malloc()/free(). Any reason why this would be a bad idea?
- NFS server often drops tcp connections after some timeout. I've seen rpc_set_autoreconnect()/etc in the source code, but just to make sure -- does libnfs auto-reconnect int this case? How long it'll keep trying? (e.g. if NFS server goes down) How long it waits between reconnect attempts?
NFS servers will often tear down idle tcp connections. Often after something like 15 minutes or so. If you have auto reconnect enabled then libnfs will automatically reconnect to the server completely transparent to the application.
You can use rpc|nfs_set_autoreconnect() to control this behaviour. <0 means, retry forever, never give up 0 means disabled. No reconnect attempts will be made.
0 means to try reconnecting that many times and then give up if that fails.
Most people either want to set it to <0 or 0.
0 is mostly useful if you want to control the reconnect process yourself. For example if when a server goes down, you do some magic then reconnect to a different server/ip. Useful in certain types of clusters.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/sahlberg/libnfs/issues/241#issuecomment-367205794, or mute the thread https://github.com/notifications/unsubscribe-auth/AAeNkJx4u0FFnhd45TOd6EyEMFvuzRv6ks5tW5OOgaJpZM4R9x_S .
Don't use UDP.
Got it. Wanted to clear this up -- it jumped out when I was reading code. (I don't use NFS over UDP anyway)
- For tcp case it makes sense (if outqueue is empty) to send data out immediately (and add remainder, if any, into outqueue) -- this would improve latency. Would you agree?
I don't think it should have much/any impact on performance unless you are completely singlethreaded.
Well, I was thinking that it is possible that my event loop has a lot of work queued right now and amount of time before even loop spins again could be unpredictable -- all this time request will be sitting in memory.
The simplest solution if you need to shave off this latency is just manually calling *_service(POLLOUT) immediately after you have called the _async() function.
Yep, I didn't think about this -- it is a better approach.
If you have auto reconnect enabled then libnfs will automatically reconnect to the server completely transparent to the application.
Questions:
- Is it on by default?
- Is there a pause between attempts? (if not -- it can cause a surge of traffic, if there is no one listening on target port)
For example if when a server goes down, you do some magic then reconnect to a different server/ip. Useful in certain types of clusters.
Isilon is probably one of them.
On Wed, Feb 21, 2018 at 5:24 PM, crusader-mike [email protected] wrote:
Don't use UDP.
Got it. Wanted to clear this up -- it jumped out when I was reading code. (I don't use NFS over UDP anyway)
- For tcp case it makes sense (if outqueue is empty) to send data out immediately (and add remainder, if any, into outqueue) -- this would improve latency. Would you agree?
I don't think it should have much/any impact on performance unless you are completely singlethreaded.
Well, I was thinking that it is possible that my event loop has a lot of work queued right now and amount of time before even loop spins again could be unpredictable -- all this time request will be sitting in memory.
The simplest solution if you need to shave off this latency is just manually calling *_service(POLLOUT) immediately after you have called the _async() function.
Yep, I didn't think about this -- it is a better approach.
If you have auto reconnect enabled then libnfs will automatically reconnect to the server completely transparent to the application.
Questions:
- Is it on by default?
It should be on by default for NFS contexts but is off by default ro RPC contexts. However, as I don't think I document or promise what the default should be, I think that if you care then you must call and set the autoretry setting explicitely.
That is the safest in case the default change by design or by bug.
- Is there a pause between attempts? (if not -- it can cause a surge of traffic, if there is no one listening on target port)
There is no pause, as I can't call sleep or similar as I want to keep the API non-blocking. Thus it can cause a surge of traffic and also make it spin in your event loop.
I don't have any good solution here :-(
Useful in certain types of clusters.
Isilon is probably one of them.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/sahlberg/libnfs/issues/241#issuecomment-367235961, or mute the thread https://github.com/notifications/unsubscribe-auth/AAeNkPVlH-G59fT86XMkHen1bFftzIIcks5tW8TIgaJpZM4R9x_S .
There is no pause, as I can't call sleep or similar as I want to keep the API non-blocking. Thus it can cause a surge of traffic and also make it spin in your event loop. I don't have any good solution here :-(
My immediate reaction was to dig in the code to figure out how you deal with nfs/rpc requests timeout. It looks like you expect user to call nfs_service() every 100ms or so even if nothing happened on file descriptor. Isn't it kind of cheating? ;-). I bet this assumption can be used it to implement reconnect delay.
Did you consider forcing client code to provide necessary facilities (event and timers) via callbacks? I.e. instead of asking user to poll -- you call mechanisms provided by him as required:
nfs_init_context()takes two function pointers- one you call when you need to be called at certain time later (timer registration/cancellation)
- another one -- when you need to be called on some event on some fd (fd event registration/cancellation)
- these callbacks could look smth like this:
typedef void (*timer_service_cb)(bool install_or_remove, int timer_id, timer_cb tcb, void *data);
typedef void (*event_service_cb)(bool install_or_remove, fd, int mask, event_cb ecb, void *data);
struct nfs_context *nfs_init_context(timer_service_cb, event_service_cb);
Client will hookup these calls directly into his event loop. It is just a general idea -- I am certain this design can be improved. What do you think about it?
There are no global variables shared across the contexts so as long as each thread has its own context then you are fine.
@sahlberg, well, I just spotted static variable (salt) in rpc_init_context() -- this make that function non-thread-safe in general case...