Bug with Radicale in Windows (Filenames are too long) + suggested (yet incomplete) patch

[Binarus]

So we've found a bug in Radicale under Windows which is a show stopper, but which should be easy to fix. In the hope that somebody will read this:

The web is full of reports about the dreaded "80004005" error which often occurs when a MUA (e.g. Thunderbird) accepts an invitation and the respective event should be entered into a CalDAV calender. We are running Radicale on a Windows server and TB at the client stations, and we were affected by that problem, too. None of the solutions which we found in the web worked.

We investigated the problem ourselves and at first noticed that it depended on which software originally had created the invitation. With invitations created by Outlook, the problem appeared; with invitation created by Thunderbird, it didn't.

Next, we started Radicale in debugging mode on the Windows server and noticed that it tried to create a very long path for a temporary object whenever we tried to accept an invitation which was created by Outlook. In our case, the path was about 280 characters long.

It is well known that Windows in its default configuration still has the moronic 250 (or whatever) character limit for file paths in place, so our next action was to disable this via GPO. Unfortunately, that still wasn't the complete solution.

Further investigation showed that Python also suffers from this limit, unless the respective path is prepended with \\?\. That's probably not Python's fault; I believe it's the Windows API which needs that prefix to handle long path names correctly. Anyway, unfortunately, Radicale does not prefix the paths appropriately.

Problem

This means: Radicale under Windows does construct file name paths which are too long if not prefixed as shown above, and does not add the necessary prefix. Subsequently, creation of the respective file fails, Radicale can't enter the respective event to the respective calendar, and Radicale reports an error back to the MUA.

Suggested, incomplete patch

In Lib\site-packages\radicale\storage\multifilesystem\base.py, line 55 (in function _atomic_write) reads

with open(os.path.join(tmp_dir, name), mode, newline=newline,

Changing it to

with open('\\\\?\\' + os.path.join(tmp_dir, name), mode, newline=newline,

(that is, prepending \\?\ to the path as described above) made the problem disappear.

Why this patch works halfway, but is bad

Please note that this is my very first experience with Python. I really don't have any clue about it. Hence, that "patch" releases some pressure (our Radicale-based CalDAV calendars are in production since several weeks), but surely has its problems:

• I am not sure about the side effects.
• I am sure that there are more file handling functions which suffer from the same problem.
• I am sure that the patch breaks portability; the paths constructed that way will fail under e.g. Linux.

Therefore, we'd be very grateful if this bug can be fixed by somebody who is familiar with Python and the Radicale server.

Further details:

Relevant portion of the Radicale log before the patch (when trying to accept an invitation in TB which was created by Outlook):

[2022-03-28 10:31:58 +0200] [6664/Thread-26 (process_request_thread)] [WARNING] Bad PUT request on '/xxxxx.yyyyyyyyyyyyy/d26920fe-cda5-18c1-e2df-821b7d278ad9/040000008200E00074C5B7101A82E00800000000E09C6DAA8A42D80100000000000000001000000041DD967A5F4C3949836CB9B6725E96EC.ics': Failed to store item '040000008200E00074C5B7101A82E00800000000E09C6DAA8A42D80100000000000000001000000041DD967A5F4C3949836CB9B6725E96EC.ics' in collection 'xxxxx.yyyyyyyyyyyyy/d26920fe-cda5-18c1-e2df-821b7d278ad9': [Errno 2] No such file or directory: 'c:\\daten\\radicale\\data\\collections\\collection-root\\xxxxx.yyyyyyyyyyyyy\\d26920fe-cda5-18c1-e2df-821b7d278ad9\\.Radicale.cache\\item\\.Radicale.tmp-71aad6za\\040000008200E00074C5B7101A82E00800000000E09C6DAA8A42D80100000000000000001000000041DD967A5F4C3949836CB9B6725E96EC.ics'
Traceback (most recent call last):
  File "C:\Daten\Radicale\lib\site-packages\radicale\storage\multifilesystem\upload.py", line 40, in upload
    self._store_item_cache(href, item)
  File "C:\Daten\Radicale\lib\site-packages\radicale\storage\multifilesystem\cache.py", line 89, in _store_item_cache
    with contextlib.suppress(PermissionError), self._atomic_write(
  File "contextlib.py", line 135, in __enter__
  File "C:\Daten\Radicale\lib\site-packages\radicale\storage\multifilesystem\base.py", line 54, in _atomic_write
    with open(os.path.join(tmp_dir, name), mode, newline=newline,
FileNotFoundError: [Errno 2] No such file or directory: 'c:\\daten\\radicale\\data\\collections\\collection-root\\xxxxx.yyyyyyyyyyyyy\\d26920fe-cda5-18c1-e2df-821b7d278ad9\\.Radicale.cache\\item\\.Radicale.tmp-71aad6za\\040000008200E00074C5B7101A82E00800000000E09C6DAA8A42D80100000000000000001000000041DD967A5F4C3949836CB9B6725E96EC.ics'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Daten\Radicale\lib\site-packages\radicale\app\put.py", line 224, in do_PUT
    etag = parent_item.upload(href, prepared_item).etag
  File "C:\Daten\Radicale\lib\site-packages\radicale\storage\multifilesystem\upload.py", line 42, in upload
    raise ValueError("Failed to store item %r in collection %r: %s" %
ValueError: Failed to store item '040000008200E00074C5B7101A82E00800000000E09C6DAA8A42D80100000000000000001000000041DD967A5F4C3949836CB9B6725E96EC.ics' in collection 'xxxxx.yyyyyyyyyyyyy/d26920fe-cda5-18c1-e2df-821b7d278ad9': [Errno 2] No such file or directory: 'c:\\daten\\radicale\\data\\collections\\collection-root\\xxxxx.yyyyyyyyyyyyy\\d26920fe-cda5-18c1-e2df-821b7d278ad9\\.Radicale.cache\\item\\.Radicale.tmp-71aad6za\\040000008200E00074C5B7101A82E00800000000E09C6DAA8A42D80100000000000000001000000041DD967A5F4C3949836CB9B6725E96EC.ics'
[2022-03-28 10:31:58 +0200] [6664/Thread-26 (process_request_thread)] [DEBUG] Response content:
Bad Request
[2022-03-28 10:31:58 +0200] [6664/Thread-26 (process_request_thread)] [INFO] PUT response status for '/xxxxx.yyyyyyyyyyyyy/d26920fe-cda5-18c1-e2df-821b7d278ad9/040000008200E00074C5B7101A82E00800000000E09C6DAA8A42D80100000000000000001000000041DD967A5F4C3949836CB9B6725E96EC.ics' in 20.765 seconds: 400 Bad Request

As we can see, Radicale constructs a very long path name which Python / Windows then can't create, and thus fails. We can also see that the length of the path name is mainly due to the long invitation / event UID, which obviously is taken as a part of the temporary file name which is constructed by Radicale.

Finally, that UID might explain why the problem depends on which MUA has created the invitation. Outlook obviously creates very long UIDs (in this case, 112 characters), while other MUAs (notably, TB) might create shorter ones. Shorter UIDs mean shorter path names, so the standard path name length limit may strike with invitations from Outlook, but may not strike with invitations from TB.

In every case, the problem is clearly due to a bug in Radicale. The file handling functions should be altered so that only under Windows every path / file name, whether for reading or writing, is prepended by \\?\.

I am looking forward to any comments :-)

Regards, and thank you very much in advance!

[Unrud]

See: Enable Long Paths in Windows 10, Version 1607, and Later

[Binarus]

Thanks for the comment.

However, the key point here is that it is not sufficient to just enable long path names in Windows. Python (or the Windows APIs it calls) still fail after having done this when trying to use long path names.

It works correctly only if the path is prepended with the string \\?\. Prefixing the path is Radicale's job, but it doesn't do it. It passes the path without that prefix to Python's open function. That is, Radicale currently does open(os.path.join(tmp_dir, name), ..., but (under Windows) should do something like open('\\\\?\\' + os.path.join(tmp_dir, name), ... (see my proposed incomplete patch).

In other words, even after having enabled long path names in Windows, Python's open(some_long_name, ...) will fail, unless some_long_name literally starts with \\?\. Radicale does not take care of that.

As described in my first post, changing the code in line 55 of base.py accordingly solved our problem, which additionally proves what I have said above.

However, I am not sure about possible side effects (yet optimistic), and I am quite sure that there are other places in the program code of Radicale which must be changed in the same way; unfortunately, I have no clue where those places may be or how I can spot them. Plus, my change clearly breaks compatibility with OSes other than Windows.

So I'd like to ask the Radicale developers whether they could implement the patch correctly / completely.

Thank you very much again, and best regards.

[Unrud]

However, the key point here is that it is not sufficient to just enable long path names in Windows. Python (or the Windows APIs it calls) still fail after having done this when trying to use long path names.

It works fine for me. Prepending \\?\ is only required when LongPathsEnabled is not set. Are you sure you enabled it and the setting is actually supported by your Windows version.

However isn't it enough to add \\?\ to the filesystem_folder setting.

[Binarus]

Thank you very much for caring!

It works fine for me. Prepending \?\ is only required when LongPathsEnabled is not set. Are you sure you enabled it and the setting is actually supported by your Windows version.

Only setting LongPathsEnabled did not work for us. However, we're 100% sure that the Windows version in question supports it. It is a Windows Server 2019 with up-to-date patches. We have set LongPathsEnabled via Group Policy Editor (where it is called Enable Win32 long paths) and then have issued gpupdate /force.

You wrote that it worked fine for you. Did you try with an invitation which was created by Outlook? In our case, that made Radicale try to create the following file: c:\\daten\\radicale\\data\\collections\\collection-root\\xxxxx.yyyyyyyyyyyyy\\d26920fe-cda5-18c1-e2df-821b7d278ad9\\.Radicale.cache\\item\\.Radicale.tmp-71aad6za\\040000008200E00074C5B7101A82E00800000000E09C6DAA8A42D80100000000000000001000000041DD967A5F4C3949836CB9B6725E96EC.ics That name obviously was too long ...

However isn't it enough to add \?\ to the filesystem_folder setting.

This is an excellent idea which I hadn't thought of yet. Right now, I don't have time to test it, but I'll do it this afternoon, and I'll report the result of course.

Thanks again!