Backup for account failed with error closed stream

[stefangries]

With one of my accounts, the backup always aborts - every time I try. The problem always occurs after about the same amount of time. Here is the error:

E, [2024-01-27T05:53:31.561444 #1] ERROR -- : Backup for account '[email protected]' failed with error closed stream
/usr/local/lib/ruby/3.2.0/openssl/buffering.rb:354:in `syswrite'
/usr/local/lib/ruby/3.2.0/openssl/buffering.rb:354:in `do_write'
/usr/local/lib/ruby/3.2.0/openssl/buffering.rb:453:in `print'
/usr/local/bundle/gems/net-imap-0.4.9.1/lib/net/imap.rb:2809:in `put_string'
/usr/local/bundle/gems/net-imap-0.4.9.1/lib/net/imap.rb:2781:in `block in send_command'
/usr/local/lib/ruby/3.2.0/monitor.rb:202:in `synchronize'
/usr/local/lib/ruby/3.2.0/monitor.rb:202:in `mon_synchronize'
/usr/local/bundle/gems/net-imap-0.4.9.1/lib/net/imap.rb:2776:in `send_command'
/usr/local/bundle/gems/net-imap-0.4.9.1/lib/net/imap.rb:2858:in `block in fetch_internal'
/usr/local/lib/ruby/3.2.0/monitor.rb:202:in `synchronize'
/usr/local/lib/ruby/3.2.0/monitor.rb:202:in `mon_synchronize'
/usr/local/bundle/gems/net-imap-0.4.9.1/lib/net/imap.rb:2853:in `fetch_internal'
/usr/local/bundle/gems/net-imap-0.4.9.1/lib/net/imap.rb:2080:in `uid_fetch'
/usr/local/lib/ruby/3.2.0/forwardable.rb:240:in `uid_fetch'
/app/lib/imap/backup/client/automatic_login_wrapper.rb:28:in `method_missing'
/app/lib/imap/backup/account/folder.rb:94:in `block in fetch_multi'
/app/lib/imap/backup/retry_on_error.rb:18:in `retry_on_error'
/app/lib/imap/backup/account/folder.rb:93:in `fetch_multi'
/app/lib/imap/backup/flag_refresher.rb:30:in `refresh_block'
/app/lib/imap/backup/flag_refresher.rb:20:in `block in run'
/app/lib/imap/backup/flag_refresher.rb:19:in `each'
/app/lib/imap/backup/flag_refresher.rb:19:in `each_slice'
/app/lib/imap/backup/flag_refresher.rb:19:in `run'
/app/lib/imap/backup/account/folder_backup.rb:60:in `clean_up'
/app/lib/imap/backup/account/folder_backup.rb:36:in `run'
/app/lib/imap/backup/account/backup.rb:35:in `block in run'
/app/lib/imap/backup/account/backup_folders.rb:42:in `block in each'
/app/lib/imap/backup/account/backup_folders.rb:42:in `each'
/app/lib/imap/backup/account/backup_folders.rb:42:in `each'
/app/lib/imap/backup/account/backup.rb:34:in `run'
/app/lib/imap/backup/cli/backup.rb:36:in `block in run'
/app/lib/imap/backup/cli/backup.rb:34:in `each'
/app/lib/imap/backup/cli/backup.rb:34:in `run'
/app/lib/imap/backup/cli.rb:94:in `backup'
/usr/local/bundle/gems/thor-1.3.0/lib/thor/command.rb:28:in `run'
/usr/local/bundle/gems/thor-1.3.0/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/bundle/gems/thor-1.3.0/lib/thor.rb:527:in `dispatch'
/usr/local/bundle/gems/thor-1.3.0/lib/thor/base.rb:584:in `start'
/app/lib/imap/backup/cli.rb:69:in `start'
/app/bin/imap-backup:16:in `block in '
/app/lib/imap/backup/logger.rb:58:in `sanitize_stderr'
/app/bin/imap-backup:15:in `'

Serverside Mailcow/Dovecot:

Jan 27 05:53:48 fe4f90b52719 dovecot: imap([email protected])<65397><5fK2B+YPPJisFgH4>: Disconnected: Logged out in=2235 out=22086 deleted=0 expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=0 body_bytes=0

[joeyates]

Hi @stefangries

It's seems to be failing after login.

It looks like you're using the container here, but below I'll write the commands as though you had imap-backup installed locally. You'll need to prepend the Docker invocation.

Please try 2 things:

1. Run backup with double verbose to see in IMAP logging shows up anything more:

imap-backup backup --accounts [email protected] --verbose --verbose --config foo.json

2. Try another commands that contact the server, e.g.:

imap-backup remote folders  [email protected] --verbose --verbose --config foo.json

[stefangries]

I have tried everything you have written. No errors occurred. Then I went back to my initial command. No more errors either. I don't get it - but that's how it is.

But the next time I ran it, the error was there again. I think I will run your verbose commands a few more times and hopefully collect error messages.

[stefangries]

Just to make sure I'm understanding this correctly: When using --verbose --verbose the string "failed with error closed stream" should be present in the log as well, right? The output is too long to check it by hand, so I just searched it for this string.

[joeyates]

@stefangries Any updates on this?

[stefangries]

Actually, I'm still on debugging this.

[joeyates]

@stefangries I'll close for now. If you discover any new info, feel free to repoen.

[ylluminate]

Hi @joeyates - experiencing a similar issue to what was described in this thread. An initial backup seems to have gone through and this is just a sync run. The error message and stack trace are almost identical. Here are the details of my setup:

• OS: macOS 14.5 (Homebrew)
• Ruby Version: 3.3.4
• IMAP Server: imap.dreamhost.com
• imap-backup Version: 15.0.2
• mirror_mode: true
• multi_fetch_size: 10
• reset_seen_flags_after_fetch: true
• download_strategy: delay_metadata

The error occurs very intermittently / unreliably, making it difficult to pinpoint the cause. Here is an excerpt from my log:

E, [2024-07-23T15:22:30.412735 #9736] ERROR -- : Backup for account '[email protected]' failed with error closed stream
/opt/homebrew/Cellar/ruby/3.3.4/lib/ruby/3.3.0/openssl/buffering.rb:351
syswrite' /opt/homebrew/Cellar/ruby/3.3.4/lib/ruby/3.3.0/openssl/buffering.rb:351:in do_write'
/opt/homebrew/Cellar/ruby/3.3.4/lib/ruby/3.3.0/openssl/buffering.rb:450
`print'
...

I have tried increasing verbosity as suggested, but I don't see anything more helpful. Wondering if there's a way to add logging to catch more details here...

Any additional thoughts or suggestions would be appreciated.