Regression in v7.2.4: creating new (sub)folders fails

[ojkastl]

General informations

• system/distribution (with version): Mac OSX
• offlineimap version (offlineimap -V): v7.2.4 (commit 06ed00a) or any later commit
• Python version: Python 2.7.17
• CLI options:

python /path/to/offlineimap/offlineimap.py -u ttyui -a somename -c /path/to/offlineimaprc 

Configuration file offlineimaprc

Basically a large one-way-sync configuration, that works with v7.2.3 and fails on v7.2.4 and later. If needed, I can provide details later.

Logs, error

ERROR: Exceptions occurred during the run!
ERROR: Creating folder INBOX.FOO.Mailinglisten.TEST5 on repository TARGET_REPO
  Folder 'INBOX'[TARGET_REPO] could not be created. Server responded: ('NO', ['INBOX already exists!'])

Traceback:
  File "/path/to/offlineimap/offlineimap/repository/Base.py", line 210, in sync_folder_structure 
    local_repo.makefolder(local_name)
  File "/path/to/offlineimap/offlineimap/repository/IMAP.py", line 562, in makefolder
    self.makefolder_single(folder_path)
  File "/path/to/offlineimap/offlineimap/repository/IMAP.py", line 580, in makefolder_single
    OfflineImapError.ERROR.FOLDER)

ERROR: Folder 'INBOX'[TARGET_REPO] could not be created. Server responded: ('NO', ['INBOX already exists!'])

Traceback:
  File "/path/to/offlineimap/offlineimap/accounts.py", line 293, in syncrunner
    self.__sync()
  File "/path/to/offlineimap/offlineimap/accounts.py", line 372, in __sync
    remoterepos.sync_folder_structure(localrepos, statusrepos)
  File "/path/to/offlineimap/offlineimap/repository/Base.py", line 210, in sync_folder_structure 
    local_repo.makefolder(local_name)
  File "/path/to/offlineimap/offlineimap/repository/IMAP.py", line 562, in makefolder
    self.makefolder_single(folder_path)
  File "/path/to/offlineimap/offlineimap/repository/IMAP.py", line 580, in makefolder_single
    OfflineImapError.ERROR.FOLDER)

Steps to reproduce the error

• Checkout offlineimap git repo to /path/to/offlineimap/offlineimap.py
• checkout v7.2.3
• create a new folder (that was not synced yet)
• run offlinemap and the sync succeeds
• checkout v7.2.4 (or any later commit/release)
• create a new folder (that was not synced yet)
• run offlinemap and the sync FAILS
• checkout v7.2.3
• run offlinemap and the sync succeeds

[nicolas33]

Please, upgrade to v7.3.2.

[ojkastl]

Please, upgrade to v7.3.2.

Sorry if my description was a little unclear, but this regression was introduced with v7.2.4 but is present in any later releases, including v7.3.2.

[ojkastl]

git bisect tells me that this was introduced with commit 57a43f5bbaf5690e029c89e378aba3ab54a0441b

[chris001]

See my comment on https://github.com/OfflineIMAP/offlineimap/commit/57a43f5bbaf5690e029c89e378aba3ab54a0441b for a fix to this issue #645 Please test it out @ojkastl and post back your results!

[nicolas33]

Make a PR with the patch, please!

[chris001]

Pull request created!

[nicolas33]

Applied. @ojkastl Could you test the current 'next' branch, please?

[ojkastl]

Sorry, but this does not work:

 ERROR: Creating folder INBOX.FOO.Mailinglisten.TEST11 on repository TARGET_REPO
  Folder 'INBOX.FOO'[TARGET_REPO] could not be created. Server responded: ('NO', ['Cannot create this folder.'])
 ERROR: Folder 'INBOX.FOO'[TARGET_REPO] could not be created. Server responded: ('NO', ['Cannot create this folder.'])
 *** Finished account 'somename in 0:01
ERROR: Exceptions occurred during the run!
ERROR: Creating folder INBOX.FOO.Mailinglisten.TEST11 on repository TARGET_REPO
  Folder 'INBOX.FOO'[TARGET_REPO] could not be created. Server responded: ('NO', ['Cannot create this folder.'])

Traceback:
  File "/path/to/offlineimap/offlineimap/repository/Base.py", line 210, in sync_folder_structure
    local_repo.makefolder(local_name)
  File "/path/to/offlineimap/offlineimap/repository/IMAP.py", line 562, in makefolder
    self.makefolder_single(folder_path)
  File "/path/to/offlineimap/offlineimap/repository/IMAP.py", line 581, in makefolder_single
    OfflineImapError.ERROR.FOLDER)

ERROR: Folder 'INBOX.FOO'[TARGET_REPO] could not be created. Server responded: ('NO', ['Cannot create this folder.'])

[chris001]

@ojkastl Just for our information: what are the IMAP server softwares and versions (if you have them) of the two IMAP servers you're sync'ing your mailboxes between?

[chris001]

@ojkastl Does the folder 'INBOX.FOO' on the [TARGET_REPO] already exist? What's the IMAP software on the Target Repo?

[ojkastl]

@ojkastl Just for our information: what are the IMAP server softwares and versions (if you have them) of the two IMAP servers you're sync'ing your mailboxes between?

Source server is GMX.net, no idea what software they run. Target server seems to be Exim.

[ojkastl]

@ojkastl Does the folder 'INBOX.FOO' on the [TARGET_REPO] already exist?

INBOX.FOO and INBOX.FOO.Mailinglisten are already existing, only INBOX.FOO.Mailinglisten.TEST11 is to be created.

[ojkastl]

The error message is the same, if I create the new folder directly in INBOX.FOO, on the same level as INBOX.FOO.Mailinglisten

[chris001]

Source server is GMX.net, no idea what software they run. Target server seems to be Exim.

Exim isn't IMAP software, it's SMTP software, made for sending email only. The most popular IMAP server software are Dovecot, Courier, and Cyrus, plus the commercial IMAP softwares Gmail, Outlook, Yahoo, GMX, etc. Here's a list of IMAP software, yours is probably on the list?: https://en.wikipedia.org/wiki/List_of_mail_server_software

[chris001]

@nicolas33 Should we add a simple function to detect the name of the IMAP mail server software, as displayed in the IMAP banner string, after the capability? This would save a lot of time, bypassing asking users what's the IMAP software their server is running, especially when many users don't know and don't really mind about which IMAP server software is running on the source and target. Because it appears that many IMAP server software gives unique text error messages that don't pattern match what we're expecting to receive from the top 5 IMAP server softwares representing probably 90% of the market.

[chris001]

@ojkastl Can you run these commands?

telnet targetrepoimapserver 143

or

openssl s_client -connect targetrepoimapserver:993

...and post the line where it says OK [CAPABILITY ...] followed by the name of the IMAP server software, for example: * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN] Dovecot ready.

[nicolas33]

Should we add a simple function to detect the name of the IMAP mail server software, as displayed in the IMAP banner string, after the capability?

That's possible. The legacy approach is to handle any case without targeting the server.

[nicolas33]

The last error is not the same. Something else is going on. It's hard to guess what's going on, though. Do you have access to the server logs?

[ojkastl]

Sorry, must have misread that:

Source server:

* OK [CAPABILITY IMAP4rev1 CHILDREN ENABLE ID IDLE LIST-EXTENDED LIST-STATUS LITERAL- MOVE NAMESPACE SASL-IR SORT SPECIAL-USE THREAD=ORDEREDSUBJECT UIDPLUS UNSELECT WITHIN STARTTLS LOGINDISABLED] IMAP server ready H migmx003 15.4 IMAP-1M8idH-1ivSVF1qgT-005JZm

Target server:

* OK [CAPABILITY IMAP4 IMAP4rev1 STARTTLS UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION] perdition ready on ::ffff:192.168.0.11 00041cfb

[ojkastl]

The last error is not the same. Something else is going on. It's hard to guess what's going on, though. Do you have access to the server logs?

Unfortunately not.

[chris001]

Should we add a simple function to detect the name of the IMAP mail server software, as displayed in the IMAP banner string, after the capability?

That's possible. The legacy approach is to handle any case without targeting the server.

It's not to run the algorithm differently per server, it's to know each server's unique way of saying its error messages in plain natural language, for example here with this Perdition IMAP server, it's not using one of the standard response codes, it's giving a response in ordinary English, Cannot create this folder, which doesn't contain any of the 2 keywords we're searching for in [ALREADYEXISTS] or Folder already exists!

[ojkastl]

Any news on this?

[nicolas33]

Could you provide the logs of the failure with the fixed version (from the 'next' branch)?

[ojkastl]

Could you provide the logs of the failure with the fixed version (from the 'next' branch)?

Sure, I just did not know there was a fixed version already. Would have been nice to see some info in this issue...

[ojkastl]

 ERROR: Creating folder INBOX.FOO.Mailinglisten.TEST01_20200405 on repository TARGET_REPO
  Folder 'INBOX.FOO'[TARGET_REPO] could not be created. Server responded: ('NO', ['Cannot create this folder.'])
 ERROR: Folder 'INBOX.FOO'[TARGET_REPO] could not be created. Server responded: ('NO', ['Cannot create this folder.'])
 *** Finished account 'FOOXXX in 0:02
ERROR: Exceptions occurred during the run!
ERROR: Creating folder INBOX.FOO.Mailinglisten.TEST01_20200405 on repository TARGET_REPO
  Folder 'INBOX.FOO'[TARGET_REPO] could not be created. Server responded: ('NO', ['Cannot create this folder.'])
  
Traceback:
  File "/path/to/offlineimap/offlineimap/repository/Base.py", line 210, in sync_folder_structure 
    local_repo.makefolder(local_name)
  File "/path/to/offlineimap/offlineimap/repository/IMAP.py", line 562, in makefolder
    self.makefolder_single(folder_path)
  File "/path/to/offlineimap/offlineimap/repository/IMAP.py", line 581, in makefolder_single
    OfflineImapError.ERROR.FOLDER)
  
ERROR: Folder 'INBOX.FOO'[TARGET_REPO] could not be created. Server responded: ('NO', ['Cannot create this folder.'])
  
Traceback:
  File "/path/to/offlineimap/offlineimap/accounts.py", line 293, in syncrunner
    self.__sync()
  File "/path/to/offlineimap/offlineimap/accounts.py", line 375, in __sync
    remoterepos.sync_folder_structure(localrepos, statusrepos)
  File "/path/to/offlineimap/offlineimap/repository/Base.py", line 210, in sync_folder_structure 
    local_repo.makefolder(local_name)
  File "/path/to/offlineimap/offlineimap/repository/IMAP.py", line 562, in makefolder
    self.makefolder_single(folder_path)
  File "/path/to/offlineimap/offlineimap/repository/IMAP.py", line 581, in makefolder_single
    OfflineImapError.ERROR.FOLDER)

Tested on next branch at commit

commit da69fd81edac89a0eabf13ffda415dd8e22e4d4c (HEAD -> next, origin/next)
Author: Frank LENORMAND <[email protected]>
Date:   Mon Mar 23 09:02:30 2020 +0300

    export env. variables when running account hooks

[chris001]

We should probably detect both "cannot" and "create", and "already" and "exists", and "NO". @nicolas33 Your ideas?

[nicolas33]

That's yet another error message. This explains the error.

We should probably detect both "cannot" and "create", and "already" and "exists", and "NO"

At this point, I'm in favour to collect the error messages and build a list in order to match the "exact" message rather than trying to guess patterns.

[ajsutton]

For the record with outlook.live.com, the response I get when the directory already exists is:

Server responded: ('NO', [b'Mailbox already exists.'])

I had to hack it into the code to get sync to work.

Rather than trying to decode the error message, is it possible to just ask if a directory with the required name and consider it created if it does exist? Won't help with case-insensitivity kind of problems but will avoid problems Iike I'm having where the folder exists with exactly the same name but offlineimap is trying to create it and then failing.