laravel-imap icon indicating copy to clipboard operation
laravel-imap copied to clipboard

Chunk with 0 messages

Open bilogic opened this issue 2 years ago • 1 comments

Describe the bug Chunk returns with 0 messages when there are still >1000 messages on server

To Reproduce Steps to reproduce the behavior: Connecting to gmail, $chunk_size and $start_chunk set to 1 The issue remains even if $chunk_size is set to 10

$i  = 0;
$folder->query()->all()->chunked(function ($messages, $chunk) use (&$i) {
    echo "- chunk #$chunk, ".$messages->count().' messages';
    $messages->each(function (Message $message) use (&$i) {
        file_put_contents(storage_path("messages/$i.xml"), $message->getRawBody());
        $i++;

        if ($message->move('INBOX/Downloaded') == true) {
            echo "- Moved {$message->uid}").PHP_EOL;
        } else {
            echo "- Cannot move {$message->uid}".PHP_EOL;
        }
    });
}, $chunk_size = 1, $start_chunk = 1);

Expected behavior There should always be >0 messages in each chunk, never 0 messages.

Screenshots

- chunk #1, 1 messages
- Moved 6003
- chunk #2, 1 messages
- Moved 11
- chunk #3, 1 messages
- Moved 20
- chunk #4, 1 messages
- Moved 27
- chunk #5, 1 messages
- Moved 31
- chunk #6, 1 messages
- Moved 35
- chunk #7, 1 messages
- Moved 38
- chunk #8, 1 messages
- Moved 40
- chunk #9, 1 messages
- Moved 42
- chunk #10, 1 messages
- Moved 44
- chunk #11, 1 messages
- Moved 46
- chunk #12, 1 messages
- Moved 48
- chunk #13, 0 messages
- chunk #14, 0 messages
- chunk #15, 0 messages
- chunk #16, 0 messages
- chunk #17, 0 messages
- chunk #18, 0 messages
- chunk #19, 0 messages
- chunk #20, 0 messages

Desktop / Server (please complete the following information):

  • PHP version: 7.4.28
  • Laravel version: 8.83.10
  • Nova version: 3.32.0
  • MySQL version: 8.0.29-0ubuntu0.21.10.2
  • MySQL Client version: mysqlnd 7.4.28
  • databaseEngine version: mysql
  • databaseVersion2 version: 8.0.29-0ubuntu0.21.10.2

Additional context

  1. I want to download and save raw messages one by one without it becoming a memory hog.
  2. Once I kill the script and restart, it downloads another 12 to 13 messages before returning 0 messages again, so it doesn't seem to be a rate limiting issue

bilogic avatar May 09 '22 09:05 bilogic

The problem is: when you move email - first item from collection removed too. Its like iterate array and remove elements.

Workaround for me - iterate 1 by 1, and request every time first element

/** @var WhereQuery $query */
$totalMessages = $this->inboxFolder->messages()->all()->count();
for ($i = 1; $i <= $totalMessages; $i++) {
   /** @var MessageCollection $messages */
   $messages = $this->inboxFolder->messages()->all()->limit(1, 1)->get();
   foreach ($messages as $message) {
      $message->move('...'); // and so on
   } 
}

Tahiaji avatar Dec 08 '22 15:12 Tahiaji