idle function not returning any message when new mail comes

[syedajmal1998]

idle function not returning any message when new mail comes

Config:

        'custom' => [
            'host' => 'outlook.office365.com',
            'port' => 993,
            'encryption' => "ssl",
            'validate_cert' => true,
            'username' => 'email@my_domain.com',
            'password' => get_access_token(),
            'authentication' => "oauth",
        ],
// halper_functions.php
function get_access_token()
    {
        $tenantId = env("TENANT_ID");
        $clientId = env("CLIENT_ID");
        $clientSecret = env("CLIENT_SECRET");
        $guzzle = new \GuzzleHttp\Client();
        $url = 'https://login.microsoftonline.com/' . $tenantId . '/oauth2/v2.0/token';
        $token = json_decode($guzzle->post($url, [
            'form_params' => [
                'client_id' => $clientId,
                'client_secret' => $clientSecret,
                'scope' => 'https://outlook.office.com/IMAP.AccessAsUser.All',
                'grant_type' => 'password',
                'username' => 'email@my_domain.com',
                'password' => 'password',
            ],
        ])->getBody()->getContents());
        $accessToken = $token->access_token;
        return $accessToken;
    }

custom command:

public function handle()
    {
        $client = Client::account("custom");
        try {
            $client->connect();
        } catch (ConnectionFailedException $e) {
            Log::error($e->getMessage());
            return 1;
        }
        try {
            /** @var Folder $folder */
            $folder = $client->getFolder("INBOX");
        } catch (ConnectionFailedException $e) {
            Log::error($e->getMessage());
            return 1;
        } catch (FolderFetchingException $e) {
            Log::error($e->getMessage());
            return 1;
        }
        try {
            dump("listening...");
            $folder->idle(function ($message) {
            dump($message); // this doesn't run
            }, 1200, true);
        } catch (ConnectionFailedException $e) {
            Log::error($e->getMessage());
            return 1;
        }

but when i get messages manually it works

Route::get("test", function () {
    $client = Client::account("custom");
    try {
        $client->connect();
    } catch (ConnectionFailedException $e) {
        throw $e;
    }
    try {
        /** @var Folder $folder */
        $folder = $client->getFolder("INBOX");
    } catch (ConnectionFailedException $e) {
        throw $e;
    } catch (FolderFetchingException $e) {
        throw $e;
    }
    
    dump($client->getFolder("INBOX")->examine());
    dump($client->getFolder("INBOX")->messages()->all()->limit(5)->get());
    dd("done");
});

• OS: CentOs (prod), win 11 (dev)
• PHP: 8.1
• webklex/laravel-imap : 2.4
• Provider: Outlook

[Webklex]

Hi @syedajmal1998 , thanks for the detailed report. I believe to have determined and fixed the issue. I'm currently testing the patch to see if it actually works as expected. I'll let you know in several hours.

It turns out the idle connection timeout / lifetime depends on the server settings and might just last for several minutes before it terminates.

The ImapProtocol class used to use the php function fgets to fetch the next line - unfortunately this method ignores the timeout set with stream_set_timeout. To fix this, I switched to reading the data stream byte by byte via fread.

I also improved the Folder::idle method to process * OK Still here callbacks received while idling. It should also "survive" temporary connection issues and always reestablish a connection.

Its currently running for about 2,5h (applied timeout: 300sec) without any issues.

Best regards,

[syedajmal1998]

thank you so much for your time I'll wait

[Webklex]

Hi @syedajmal1998 , looks like its fixed :)

Please update to the latest release: https://github.com/Webklex/php-imap/releases/tag/4.0.0 (might take a few minutes, until the release gets available via composer).

Best regards & happy coding,

[syedajmal1998]

thank you so much for time it means alot, but also can you please update webklex/laravel-imap dependencies, i cant install it

[Webklex]

You are right, its long overdue. A new release is now available: https://github.com/Webklex/laravel-imap/releases/tag/4.0.0

Best regards,

[syedajmal1998]

sorry for asking, but its still the same for me.

[syedajmal1998]

also a new bug:

Illegal offset type

at vendor\webklex\php-imap\src\Connection\Protocols\ImapProtocol.php:646 642▕ 643▕ return $data; 644▕ } 645▕ if ($uid) { ➜ 646▕ $result[$tokens[2][$uidKey]] = $data; 647▕ }else{ 648▕ $result[$tokens[0]] = $data; 649▕ } 650▕ }

1 vendor\webklex\php-imap\src\Connection\Protocols\ImapProtocol.php:697 Webklex\PHPIMAP\Connection\Protocols\ImapProtocol::fetch()

2 \webklex\php-imap\src\Query\Query.php:228 Webklex\PHPIMAP\Connection\Protocols\ImapProtocol::flags()

when:

$client->getFolder("INBOX")->messages()->all()->limit(5)->get()

[Webklex]

Please enable the debug mode inside your config/imap.php config file: https://github.com/Webklex/laravel-imap/blob/4.0.0/src/config/imap.php#L152

The debug output should look something like this, during the "startup" process:

>> TAG1 LOGIN "[email protected]" "some password"
<< TAG1 OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY LITERAL+ NOTIFY SPECIAL-USE QUOTA] Logged in
>> TAG2 CAPABILITY
<< * CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY LITERAL+ NOTIFY SPECIAL-USE QUOTA
<< TAG2 OK Capability completed (0.001 + 0.000 secs).
>> TAG3 SELECT "INBOX"
<< * FLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk unknown-1)
<< * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk unknown-1 \*)] Flags permitted.
<< * 136 EXISTS
<< * 0 RECENT
<< * OK [UNSEEN 94] First unseen.
<< * OK [UIDVALIDITY 1488899637] UIDs valid
<< * OK [UIDNEXT 275] Predicted next UID
<< TAG3 OK [READ-WRITE] Select completed (0.010 + 0.000 + 0.009 secs).
>> TAG4 IDLE
<< + idling

You could also try to reduce the timeout from 1200 to 300 or some other lower number.

[syedajmal1998]

<< TAG1 OK AUTHENTICATE completed.

TAG2 CAPABILITY << * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS MOVE ID UNSELECT CLIENTACCESSRULES CLIENTNETWORKPRESENCELOCATION BACKENDAUTHENTICATE CHILDREN IDLE NAMESPACE LITERAL+ << TAG2 OK CAPABILITY completed. TAG3 SELECT "INBOX" << * 1516 EXISTS << * 0 RECENT << * FLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent) << * OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)] Permanent flags << * OK [UNSEEN 21] Is the first unseen message << * OK [UIDVALIDITY 14] UIDVALIDITY value << * OK [UIDNEXT 3977] The next unique identifier value << TAG3 OK [READ-WRITE] SELECT completed. TAG4 IDLE << + IDLE accepted, awaiting DONE command.

[syedajmal1998]

here is the full one

<< * OK The Microsoft Exchange IMAP4 service is ready. [some long code]

TAG1 AUTHENTICATE XOAUTH2 [some too long code] << TAG1 OK AUTHENTICATE completed. TAG2 LIST "" "*" << * LIST (\Marked \HasNoChildren) "/" Archive << * LIST (\Marked \HasChildren) "/" Calendar << * LIST (\HasNoChildren) "/" Calendar/Birthdays << * LIST (\HasNoChildren) "/" "Calendar/United States holidays" << * LIST (\HasChildren) "/" Contacts << * LIST (\HasChildren) "/" "Conversation History" << * LIST (\Marked \HasNoChildren \Trash) "/" "Deleted Items" << * LIST (\Marked \HasNoChildren \Drafts) "/" Drafts << * LIST (\Marked \HasNoChildren) "/" INBOX << * LIST (\HasNoChildren) "/" Journal << * LIST (\HasNoChildren \Junk) "/" "Junk Email" << * LIST (\HasNoChildren) "/" Notes << * LIST (\HasNoChildren) "/" Outbox << * LIST (\HasNoChildren) "/" "RSS Subscriptions" << * LIST (\HasNoChildren \Sent) "/" "Sent Items" << * LIST (\HasChildren) "/" "Sync Issues" << * LIST (\HasNoChildren) "/" "Sync Issues/Conflicts" << * LIST (\HasNoChildren) "/" "Sync Issues/Local Failures" << * LIST (\HasNoChildren) "/" "Sync Issues/Server Failures" << * LIST (\HasNoChildren) "/" Tasks << TAG2 OK LIST completed. ^ "listening..." TAG3 LOGOUT << * BYE Microsoft Exchange Server IMAP4 server signing off. << TAG3 OK LOGOUT completed. << * OK The Microsoft Exchange IMAP4 service is ready. [again some long code] TAG1 AUTHENTICATE XOAUTH2 [again some too long code] << TAG1 OK AUTHENTICATE completed. TAG2 CAPABILITY << * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS MOVE ID UNSELECT CLIENTACCESSRULES CLIENTNETWORKPRESENCELOCATION BACKENDAUTHENTICATE CHILDREN IDLE NAMESPACE LITERAL+ << TAG2 OK CAPABILITY completed. TAG3 SELECT "INBOX" << * 1516 EXISTS << * 0 RECENT << * FLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent) << * OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)] Permanent flags << * OK [UNSEEN 21] Is the first unseen message << * OK [UIDVALIDITY 14] UIDVALIDITY value << * OK [UIDNEXT 3977] The next unique identifier value << TAG3 OK [READ-WRITE] SELECT completed. TAG4 IDLE << + IDLE accepted, awaiting DONE command.

[Webklex]

That looks exactly as expected. Now send an email to that mailbox and check the debug output. You should see something. Please let me know what happens next :)

[syedajmal1998]

i sent multiple messages but no response no new log

[Webklex]

But you did start the idle command before sending the messages correct? Which folder are you idling on? Is it "INBOX" or another folder? If you check the mailbox with another client, where are the new messages located? Inside "INBOX" or another folder?

You might also have to wait a little bit until the server has processed the mail and decides to notify all active sessions.

[syedajmal1998]

yes i started idle command before sending new mails and waited and in outlook new mails were in inbox folder

[Webklex]

Another thought: try to set the timeout to 60 and let the command run at least 120 seconds after you've send the test mail. The timeout should "kick" in during that time and try to fetch a response again. Would be interesting if that changes anything.

[syedajmal1998]

i think it worked but got this message DONE << * 1 RECENT

TAG5 IDLE << * 1521 EXISTS

Webklex\PHPIMAP\Exceptions\RuntimeException

idle failed

at vendor\webklex\php-imap\src\Connection\Protocols\ImapProtocol.php:1043 1039▕ / 1040▕ public function idle() { 1041▕ $this->sendRequest("IDLE"); 1042▕ if (!$this->assumedNextLine('+ ')) { ➜ 1043▕ throw new RuntimeException('idle failed'); 1044▕ } 1045▕ } 1046▕ 1047▕ /*

1 vendor\webklex\php-imap\src\Folder.php:401 Webklex\PHPIMAP\Connection\Protocols\ImapProtocol::idle()

2 app\Console\Commands\FetchIdleCommand.php:250 Webklex\PHPIMAP\Folder::idle(Object(Closure))

TAG6 LOGOUT << TAG4 OK IDLE completed. << + IDLE accepted, awaiting DONE command. << TAG6 BAD Command received in Invalid state.

[Webklex]

That's indeed interesting! I would have expected to see something like this:

>> TAG4 IDLE
<< + IDLE accepted, awaiting DONE command.
<< * 1521 EXISTS
>> DONE

1521 EXISTS was sent to early by the server - before IDLE was actually acknowledged to have started. I haven't seen this before. I wonder what could cause this. Perhaps this is an outlook specific feature related to "recent" messages? But this would still be strange - anyway, do you get the same error if you have no recent messages?

Btw you have uid_cache enabled or disabled inside your config? If it is disabled, please enable it. Ref.: https://github.com/Webklex/php-imap/blob/master/src/config/imap.php#L153

Best regards & thanks for helping me understand this "feature",

[syedajmal1998]

'uid_cache' was not in config

[syedajmal1998]

these are all new, Microsoft dropped support for logging in using username & password for imap and everything was working fine till 3-4 days ago. im using idle command for around 3-4 months this never happened

[Webklex]

I hacked together an alternative idle function, which doesn't check if the IDLE command got acknowledged.

/** @var \Webklex\PHPIMAP\Client $client */
$client->connect();
/** @var \Webklex\PHPIMAP\Folder $folder */
$folder = $client->getFolder("INBOX");
$timeout = 300;
$callback = function($message){
    dump("New message received: ".$message->subject);
};

/**
 * @param \Webklex\PHPIMAP\Connection\Protocols\ProtocolInterface $connection
 * @return void
 */
function idle($connection) {
    $connection->sendRequest("IDLE");
}

$client->setTimeout($timeout);
if (!in_array("IDLE", $client->getConnection()->getCapabilities())) {
    throw new NotSupportedCapabilityException("IMAP server does not support IDLE");
}
$client->openFolder($folder->path, true);
$connection = $client->getConnection();
idle($connection);

$sequence = ClientManager::get('options.sequence', IMAP::ST_MSGN);

while (true) {
    try {
        $line = $connection->nextLine();

        if (($pos = strpos($line, "EXISTS")) !== false) {
            $connection->done();
            $msgn = (int) substr($line, 2, $pos -2);

            $client->openFolder($folder->path, true);
            $message = $folder->query()->getMessageByMsgn($msgn);
            $message->setSequence($sequence);
            $callback($message);

            $event = $folder->getEvent("message", "new");
            $event::dispatch($message);
            idle($connection);
        }
    }catch (\Webklex\PHPIMAP\Exceptions\RuntimeException $e) {
        if(strpos($e->getMessage(), "empty response") >= 0 && $connection->connected()) {
            $connection->done();
            idle($connection);
            continue;
        }
        if(strpos($e->getMessage(), "connection closed") === false) {
            throw $e;
        }

        $client->reconnect();
        $client->openFolder($folder->path, true);

        $connection = $client->getConnection();
        idle($connection);
    }
}

In this snipped I could see $connection->done(); causing problems, because it might get called to early (?) But that exception should be catchable if it does indeed causes problems.

Best regards,

[syedajmal1998]

thank you so much this worked pretty well. also i set timeout to 20s bcs it waits till timeout ends for new mail to fetch not like before

[syedajmal1998]

that block of code worked and fixed my issue, if this is fixed in package you can close this issue

[syedajmal1998]

and sorry if i said or did something wrong this was my first ever issued a bug in github :)

[Webklex]

Hi @syedajmal1998 , nah you've did everything right :) Thanks for bringing this issue up :+1:

At the moment I would like to wait a little bit to see if this "feature" gets fixed by outlook / Microsoft or if other providers behave the same. I don't think they intended it to work this way. Also removing the "idle" check in general isn't great. So for now I wont modify the idle methods. However you can use the snipped from above to build your own custom idle implementation.

I hope you're having a great weekend :)

Best regards,

[Webklex]

Please update to v5.1 and give it another try. If you are currently using an older version below v5.0, please read the breaking changes leading up to v5.1 before upgrading.

Best regards,

[repo-dev-1]

Hello Weblex Thanks for your work. But this problem is still relevant. I can’t figure out what exactly is the reason, but idle always crashes with an “empty response” after the time set in the timeout. (Same situation on different mail servers)

[Webklex]

Hi @repo-dev-1 , please create a new issue and provide as much information as possible.

Best regards & happy coding,