idle_done() Hangs When Network Connection Severed

[TheConcolor]

While trying to find a good way to detect when there is no network connection while in a idle_check() loop, I discovered that if the network connection to the IMAP server is severed after idle() has been called, a call to idle_close() just hangs unless the network connection is restored.

with IMAPClient(host) as server:
    server.login(address, password)
    
    select = server.select_folder("INBOX", readonly=True)
    
    while True:

        i = 0

        print("IDLE starting...")
        server.idle()
        print("IDLE started")

        # Reset IDLE state every 30 seconds to ensure IMAP server connection exists
        while i < 6:
            responses = server.idle_check(timeout=5)
            print("Server sent:", responses if responses else "nothing")
            i += 1
        
        print("Stopping IDLE...")
        server.idle_done()
        print("IDLE stopped")

The purpose of stopping the IDLE state and restarting it every 30 seconds is to ensure a connection to the IMAP server still exists since idle_check() does not detect a connection loss.

Steps to Reproduce

1. Connect to IMAP server and call idle()
2. Disable network connection on client (I did this by disabling my wired NIC in NetworkManager)
3. Code hangs at server.idle_done() while no network connection

Code Output

IDLE starting...
IDLE started
Server sent: nothing
Server sent: nothing
Server sent: nothing      # NIC disbled at this point
Server sent: nothing
Server sent: nothing
Server sent: nothing
Stopping IDLE...          # Code hangs here unless network connection restored

My Setup

• Linux Mint 20
• Python v3.8.10
• IMAPClient v2.2.0 (installed via pip)

[mjs]

Thanks for the detailed report. I’ll take a look.

[TheConcolor]

If you need any more information for testing, just let me know. Thanks for taking a look!

[BoniLindsley]

I can reproduce this issue, and the traceback suggests that it was stuck waiting for a response in line 975 from idle_done: https://github.com/mjs/imapclient/blob/0279592557495d4ddf7619b17ed9e73b21161bdf/imapclient/imapclient.py#L973-L975

This means the DONE request was sent "successfully" and it then waited on a disconnected socket. This makes sense because, with so little data, sending only fills the network buffer, and receiving can block indefinitely by default.

I believe the receiving part is expected behaviour that can be configured with IMAPClient(HOST, timeout=TIMEOUT). This blocking behaviour can be reproduced with select_folder as well.

I am uncertain whether DONE succeeding should be considered expected though.

[mjs]

Sorry for my slow responses. I don't have much spare time these days.

My guess would be that if the server kills the connection such that a TCP reset arrives at the client, the read should fail because the client OS will terminate the connection. If packets between the server and client are just dropped then I could see idle_done getting stuck (and other commands too). This is somewhat expected. When packets are dropped it is impossible to distinguish a slow connection from a broken one.

Using a socket timeout is probably the best approach to deal with this kind of thing.

Can you think of anything that IMAPClient should be doing differently? Maybe a generous timeout should be the default?

[BoniLindsley]

Using a socket timeout is probably the best approach to deal with this kind of thing.

So, like a timeout parameter to override on idle_done? Or expose the _timeout attribute for future requests?

Can you think of anything that IMAPClient should be doing differently? Maybe a generous timeout should be the default?

Nothing really comes to mind in terms of changing behaviour. And I am not sure there is a sensible default, with varying use cases. I would suggest a "no defeault" instead, to force the user to acknowledge that they have picked something, but that breaks existing code.

As for what can be changed... I am not sure. The only thing that comes to mind is, may be remind the user of the timeout in the documentation of idle_done? Or may be in "advanced usage" next to the IDLE example? I can imagine the example being used a lot as template code.