Upgrade from radicale-18e4677 to 3.0.6: No access to calendars

[pawsey-kbuckley]

Have been running a radicale-18e4677 instance, which has been accessed via Thunderbird.

The only active line in the config file has been in the storage stanza

filesystem_folder = /path/to/Radicale/

so basically then, running with all the defaults, noting, in reference to this issue

[auth]

# Authentication method
# Value: none | htpasswd | remote_user | http_x_remote_user
#type = none

although I have been invoking Radicale on the command line with

--host=127.0.0.1:5232 --debug

Access to the calendars has merely required typing in my username and any string, as a password, into the Thunderbird password dialog.

All been working fine.

Have installed Radicale 3.0.6

Seems happy to startup against the existing config file

Can't seem to access any calendars though.

Running verify-storage does see some things changed, as in, it creates a few

.Radicale.cache

files and updates some

.Radicale.cache/history/

files.

Still can't access any calendars though.

In the log, I see lines such as

[2021-01-25 10:08:19 +0800] [12025/Thread-14] [DEBUG] Sanitized script name: ''
[2021-01-25 10:08:19 +0800] [12025/Thread-14] [DEBUG] Sanitized path: '/pawsey/'
[2021-01-25 10:08:19 +0800] [12025/Thread-14] [INFO] Successful login: 'username'
[2021-01-25 10:08:19 +0800] [12025/Thread-14] [DEBUG] Request content:
<?xml version="1.0"?>
<propfind xmlns="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav" xmlns:CS="http://calendarserver.org/ns/">
  <prop>
    <resourcetype />
    <owner />
    <current-user-principal />
    <supported-report-set />
    <C:supported-calendar-component-set />
    <CS:getctag />
  </prop>
</propfind>

[2021-01-25 10:08:19 +0800] [12025/Thread-14] [INFO] Access to '/pawsey/' denied for 'username'
[2021-01-25 10:08:19 +0800] [12025/Thread-14] [DEBUG] Response content:
Access to the requested resource forbidden.
[2021-01-25 10:08:19 +0800] [12025/Thread-14] [INFO] PROPFIND response status for '/pawsey/' with depth '0' in 0.002 seconds: 403 Forbidden

Interestingly, if I run the radicale-18e4677 instance against the "verified storage" I get access as before.

Do I need to add anything to the old config file to allow for 3.0.6 ?

For completeness, when accessing with radicale-18e467 I see

[7fad4fbc7700] DEBUG: anonymous user has read and write access to collection 'pawsey'
[7fad4fbc7700] INFO: Access to '/pawsey/' denied for anonymous user
[7fad4fbc7700] DEBUG: Asking client for authentication

and, after logging in as before

[7fad4fbc7700] DEBUG: Sanitized script name: ''
[7fad4fbc7700] DEBUG: Sanitized path: '/pawsey/'
[7fad4fbc7700] INFO: Successful login: 'username'
[7fad4fbc7700] DEBUG: Request content:
<?xml version="1.0"?>
<propfind xmlns="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav" xmlns:CS="http://calendarserver.org/ns/">
  <prop>
    <resourcetype />
    <owner />
    <current-user-principal />
    <supported-report-set />
    <C:supported-calendar-component-set />
    <CS:getctag />
  </prop>
</propfind>

[7fad4fbc7700] DEBUG: 'username' has read and write access to collection 'pawsey'

[MartinSchmidt123]

There seems to be a larger differences in how authentication is handled in radicale 2 and 3 (new config file format, new permission options).

Especially, the default auth type backend changed from None in Radicale 2.1 to none in Radicale 3.0

• Radicale 2.1: None : Just allows all usernames and passwords. It also disables rights checking.
• Radicale 3.0: none : Just allows all usernames and passwords.

The difference is quite subtle: there seems to be some rights checking in 3.0. This might be what causes your problems.

The default rights backend is owner_only. You could try to change it to authenticated (Authenticated users can read and write everything). But it seems users without password are seen as anonymous users. In this case you would need to allow anonymous access to your calenders via a rights file, e.g. (untested!)

[all]
user = .*
collection = .*
permission = rw

[pawsey-kbuckley]

Thanks for the suggestion Martin.

After

[2021-02-08 15:30:18 +0800] [14519] [INFO] Starting Radicale
[2021-02-08 15:30:18 +0800] [14519] [INFO] auth type is 'radicale.auth.none'
[2021-02-08 15:30:18 +0800] [14519] [INFO] storage type is 'radicale.storage.multifilesystem'
[2021-02-08 15:30:18 +0800] [14519] [INFO] rights type is 'radicale.rights.authenticated'
[2021-02-08 15:30:18 +0800] [14519] [INFO] web type is 'radicale.web.internal'
[2021-02-08 15:30:18 +0800] [14519] [INFO] Listening on '[127.0.0.1]:5232'
[2021-02-08 15:30:18 +0800] [14519] [INFO] Radicale server ready

I am still able to login with any password, although, as I think I read others reporting, I get the dialog twice

[2021-02-08 15:39:16 +0800] [14519/Thread-9] [DEBUG] Sanitized script name: ''
[2021-02-08 15:39:16 +0800] [14519/Thread-9] [DEBUG] Sanitized path: '/pawsey/'
[2021-02-08 15:39:16 +0800] [14519/Thread-9] [INFO] Successful login: 'kbuckley'
[2021-02-08 15:39:16 +0800] [14519/Thread-9] [DEBUG] Request content:

but then I still see

[2021-02-08 15:39:16 +0800] [14519/Thread-9] [INFO] Access to '/pawsey/' denied for 'kbuckley'
[2021-02-08 15:39:16 +0800] [14519/Thread-9] [DEBUG] Response content:
Access to the requested resource forbidden.
[2021-02-08 15:39:16 +0800] [14519/Thread-9] [INFO] PROPFIND response status for '/pawsey/' with depth '0' in 0.002 seconds: 403 Forbidden

Also worth pointing out that the log doesn't explictly say that it is reading the rights file, although I assume that if it wasn't, there'd be a an error message pointing that out.

[pawsey-kbuckley]

When I set

type = from_file

I then see

[2021-02-08 16:27:44 +0800] [17084/Thread-3] [DEBUG] Sanitized script name: ''
[2021-02-08 16:27:44 +0800] [17084/Thread-3] [DEBUG] Sanitized path: '/pawsey/'
[2021-02-08 16:27:44 +0800] [17084/Thread-3] [INFO] Successful login: 'kbuckley'
[2021-02-08 16:27:44 +0800] [17084/Thread-3] [DEBUG] Rule 'kbuckley':'pawsey' matches '.*':'.*' from section 'root'
[2021-02-08 16:27:44 +0800] [17084/Thread-3] [DEBUG] Request content:

which looks good, but then, after the propfind stanza,

[2021-02-08 16:27:44 +0800] [17084/Thread-3] [DEBUG] Rule 'kbuckley':'' matches '.*':'.*' from section 'root'
[2021-02-08 16:27:44 +0800] [17084/Thread-3] [INFO] Access to '/pawsey/' denied for 'kbuckley'
[2021-02-08 16:27:44 +0800] [17084/Thread-3] [DEBUG] Response content:
Access to the requested resource forbidden.
[2021-02-08 16:27:44 +0800] [17084/Thread-3] [INFO] PROPFIND response status for '/pawsey/' with depth '0' in 0.004 seconds: 403 Forbidden

although clearly, I own all of the calendar directories below the collection-root, including the one that matches my username, for which I put in a rights stanza matching the "princiapl" one the 3.0 docs.

Hmmmm?

[pavinjosdev]

Run radicale --verify-storage. The upgrade checklist might come in handy: https://github.com/Kozea/Radicale/blob/3.0.1/NEWS.md#upgrade-checklist

[pawsey-kbuckley]

Run radicale --verify-storage. The upgrade checklist might come in handy: https://github.com/Kozea/Radicale/blob/3.0.1/NEWS.md#upgrade-checklist

So your suggesting that I need to run radicale --verify-storage more than once?

At which point in the upgrade process, other than the one that I mentioned above?

Or do I just need to keep on running it until things start working as expected?

[pavinjosdev]

@pawsey-kbuckley Run the command after upgrading to check for config & storage errors. You may run it more than once to check if everything is setup correctly. For more useful output, set the logging level to info or debug.

[m1027]

I can confirm the exact issue like the OP, and honesty I gave up and returned to 2.1.x. I am using a setup with apache / basic auth / htpasswd / wsgi / radicale. The showstopper was that Thunderbird (Lightning) could not be convinced to up-/download from/to radiacle anymore whereas other clients like vdirsyncer worked: It just continuously asked for the password. Additionally to the error messages mentioned by the OP and by giving apache auth and radicale auth different "realm" names, I noticed that in Thunderbird the password dialog comes first from apache, then radicale, too -- even if radicale's auth is disabled (none).

If anybody has success to make Thunderbird calenders work let me know.

[henning-schild]

I just took over maint in gentoo and clearly staying on old versions not maintained by upstream is not a good idea.

For anyone to look into that and fix it we need concrete steps on how to reproduce the problem. Maybe with a container and installing with pip ... so not gentoo specific, and setting up all the bits to connect a thunderbird from the host.

Or one could trace/dig deeper for those who happen to have a repo on their machines. Maybe things simply boil down to file permissions, or apparmor or what have you.

As it looks to me we have one user @pawsey-kbuckley running radicale as a service directly. Are you still interested and what is your current solution/workaround?

And we have @m1027 who seems to be running wsgi on an apache. Are we even sure both are on the same problem?

[m1027]

After a couple of hours of testing again, I successfully updated from radicale 2.1.x to 3.1.5 on Gentoo/arm64. My setup is apache + mod_wsgi + radicale which is not perfectly documented.

I'd like to share the critical pitfalls here:

(1) Config file changes / logging

Radicale changed some config file variables, and even the way how to enable logging. So, better do not stick to your 2.x config file but use the new default and adjust that for your needs. For more logging, set: level = info in the [logging] section.

(2) Python version

radicale-3.1.x finally supports python-3.9+ but mod_wsgi-4.7.x does not, so you need to upgrade mod_wsgi, too. I am using 4.9.0. If your mod_wsgi is incompatible you get errors in the logs when restarting apache but the reason is not obvious.

(3) After upgrading radicale from 2.x to 3.x

... do the usual data verfication pass:

radicale --logging-level debug --verify-storage

Do not cancel even if it seems to hang for minutes. You may need to chown -R radicale:radicale your radicale file store usually below /var/lib/radicale/.

(4) Configuring auth is important for radicale 3.x

While in radicale-2.x the dumb default was enough, in 3.x you need to set this:

[auth] type = remote_user

(5) Configuring permissions are important, too, for radicale 3.x

This is the most obscure configuration step: In radicale-2.x the permissions (called 'rights' in radicale) were not required to be configured for the apache/mod_wsgi setup at all. Now, it is necessary.

Additionally, the obvious configuration

[rights] type = authenticated

causes errors. Instead I had to configure this:

[rights] type = from_file file = /etc/radicale/rights

and also create the file /etc/radicale/rights:

[all] user: .* collection: .* permissions: rw

Maybe there are better ways to configure the rights section, And, as apache already authenticates the users I currently fail to understand why we have to configure this additionally at all (or why the default is not sufficiant).

Another hint on this: If you receive logs like these

login succeeds, username is propagated, but access denied: [wsgi:error] [INFO] Successful login: 'user1' [wsgi:error] [INFO] Access to '/user1/' denied for 'user1'

then step (4) is probably okay but you sitll have issued with step (5).

WFM, thanks.

[m1027]

I have started to do further client tests against radicale-3.1.5:

• vdirsyncer works for me
• davx5 works
• thunderbird: With issues on several boxes: TB can create calendar entries and other clients can see them but TB does not display any previous calender entries nor those newly created by other clients. Might be something trivial but obviously a show stopper for TB users. The radicale logs show that TB currently tries PROPFIND with depth = 0 while the other clients try or retry with depth = 1. So, TB does not read the actual files, just the properties. These kinds of issues have been mentioned above, too.

[m1027]

I finally made thunderbird work for me against radicale-3.x, too.

See here: https://github.com/Kozea/Radicale/issues/1222

So, I believe this issue could be closed then.

[pawsey-kbuckley]

Glad to hear that it works for you, however, my issue should not be closed yet.

Installed Radicale-3.1.5 from scratch.

Used same config file and rights as detailed before, with slight path changes for new version :

$egrep -v "^#|^$" /home/username/.config/radicale/config
[server]
[encoding]
[auth]
[rights]
type = authenticated
file = /path/to/radicale-3.1.5/rights
[storage]
filesystem_folder = /path/to/Radicale315/
[web]
[logging]
[headers]
$
$ cat /path/to/radicale-3.1.5/rights
[root]
user = .*
collection = .*
permissions = RW

[principal]
user: .*
collection: {user}
permissions: RW
$

Fire up the Radicale server.

Try to create new calendars against the localhost URIs as before,

http://127.0.0.1:5232/username/calname

both with the checkbox for

This calendar doesn't require credentials

checked and, for a second calendar, unchecked.

Tbird is happy to allow me to create the calendars, however, creating an event in either calendar results in

An error occurred when writing to the calendar <calname>!
Please see below for more information.

Error code: MODIFICATION_FAILED

If you're seeing this message after snoozing or dismissing a reminder
and this is for a calendar you do not want to add or edit events for,
you can mark this calendar as read-only to avoid such experience in
future. To do so, get to the calendar properties by right-clicking on
this calendar in the list in the calendar or task view.

so It would appear that nothing has changed from the experience with Radicale 3.0.6, indeed I can still see the

PROPFIND response status for '/username/calname' with depth '0' in 0.001 seconds: 404 Not Found

messages in the log output.

Whilst the server has created, below the filesystem_folder

drwxrwxr-x   2 username  4096 Mar  9 10:55 username/
drwxr-xr-x 384 username 12288 Mar  9 11:04 ../
drwxrwxr-x   2 username  4096 Mar  9 11:07 collection-root/
drwxrwxr-x   4 username  4096 Mar  9 11:07 ./
-rw-rw-r--   1 username     0 Mar  9 11:11 .Radicale.lock

no ICS-files are created below either of the username and collection-root directories, which remain empty.

I'll be going back to commit 18e4677.

[m1027]

Okay, yeah, I had that error message from Thunderbird, too, while doing all my tests.

So, here some more hints what may help:

(1) If you are using systemd keep in mind that the radicale.service file may also block access. For example, try to move your data into the radicale's home dir (which might be /var/lib/radicale), as systemd sets permissions on that path.

(2) Try using boring default local file paths first. In my case the default storage path is /var/lib/radicale/collections (with "s"), and radicale then creates collection-root (without "s") inside of it. Inside that, have username/calname/*.ics files.

(3) The Rights file: I recommend to set logging to debug level. Then radicale explains exactly what rule is applied, which one matches and why (not). Also try something more open (like ".*") and also "rw" (instead of "RW", see radicale's docs).

[my_rule]
user: .*
collections: .*
permissions: rw

HTH

[pawsey-kbuckley]

Appears that emails to the ticket aren't making it into the comments, so here's a recap of my original reply to the above and how I tried to comment further recently:

2022/03/09, 17:11

>> Okay, yeah, I had that error message from Thunderbird, too, while
>> doing all my tests.
>> 
>> So, here some more hints what may help:
>> 
>> (1) If you are using systemd ...

Not using it for this: never been a fan of it.

Everything, including the pythin dependencies, has been installed below
a self-contained "package" directory and it's being invkoed, by me, as:

PYTHONPATH=/path/to/radicale-3.1.5/usr/local/lib/python3.6/dist-packages/ \
   /path/to/radicale-3.1.5/usr/local/bin/radicale \
   --hosts=127.0.0.1:5232 \
   --debug

>> (2) Try using boring default local file paths first.

Can certainly give it a go but, as the Radicake process is running
as me, file paths below my home are about as boring as you can
get, from a "rights" PoV, indeed, as noted, the Radicale server
clearly HAS the ability to create its default directories, but
then bombs on creating files in the directories it has created!

>> (3) The Rights file: I recommend to set logging to debug level. Then radicale
>> explains exactly what rule is applied, which one matches and why (not). ...

Wasn't sware of a separtae rights file logging directive: cheers.
Will re-read the docs.

>> ... Also
>> try something more open (like ".*") and also "rw" (instead of "RW", see
>> radicale's docs).
>> 
>> ```
>> [my_rule]
>> user: .*
>> collections: .*
>> permissions: rw
>> ```


It'd better not the capitalisation of the permissions value,
because I'm sure I started my 3.x experience with an example
file found within the Radicale docs at the time, so that would
be really annoying.

Thanks for following up.

2022/03/18, 14:04

>>>> 
>>>> try something more open (like ".*") and also "rw" (instead of "RW", see
>>>> radicale's docs).
>>>> 
>>>> ```
>>>> [my_rule]
>>>> user: .*
>>>> collections: .*
>>>> permissions: rw
>>>> ```
>> 
>> 
>> It'd better not the capitalisation of the permissions value,
>> because I'm sure I started my 3.x experience with an example
>> file found within the Radicale docs at the time, so that would
>> be really annoying.
>> 
>> Thanks for following up.

So, this is really annoying.

For my latest testing, all I changed was the case of the RW
argument to the permissions parameter in the rights file.

I am now able to create calendars and events against 3.1.5.

WAnt to do a bit more testing before signing off as "done".