Radicale icon indicating copy to clipboard operation
Radicale copied to clipboard

Published 20 hours ago • verified publisher icon Kozea

Migrating from 2.1.12 to 3.1.8: Access to the requested resource forbidden.

Open copyrights opened this issue 2 years ago • 5 comments

I'm trying migrate from version 2.1.12 to version 3.1.8 and I'm getting Access to the requested resource forbidden.

My radicale.conf

[server]
hosts = 0.0.0.0:5232
ssl = False

[encoding]
request = utf-8
stock = utf-8

[auth]
type = htpasswd
htpasswd_filename = /data/users
htpasswd_encryption = md5
realm = Password Required

[rights]
type = owner_only

[storage]
type = multifilesystem
filesystem_folder = /data/collections

[logging]
level=debug

[headers]

The Client (DAVx5) is configured to https://mydomain.org/my/prefix/myuser/

Logs:

[2022-09-04 20:34:15 +0000] [1/Thread-63 (process_request_thread)] [INFO] PROPFIND request for '/myuser/' with depth '0' received from 172.20.0.4 (forwarded for '192.168.1.14, 172.20.0.2') using 'DAVx5/4.2.3.1-ose (2022/08/27; dav4jvm; okhttp/4.10.0) Android/12'
[2022-09-04 20:34:15 +0000] [1/Thread-63 (process_request_thread)] [DEBUG] Request headers:
{'CONTENT_LENGTH': '266',
 'CONTENT_TYPE': 'application/xml; charset=utf-8',
 'GATEWAY_INTERFACE': 'CGI/1.1',
 'HTTP_ACCEPT_ENCODING': 'gzip',
 'HTTP_ACCEPT_LANGUAGE': 'de-DE, de;q=0.7, *;q=0.5',
 'HTTP_AUTHORIZATION': 'Basic **masked**',
 'HTTP_CONNECTION': 'close',
 'HTTP_DEPTH': '0',
 'HTTP_HOST': 'webcal',
 'HTTP_USER_AGENT': 'DAVx5/4.2.3.1-ose (2022/08/27; dav4jvm; okhttp/4.10.0) '
                    'Android/12',
 'HTTP_X_FORWARDED_FOR': '192.168.1.14, 172.20.0.2',
 'HTTP_X_FORWARDED_HOST': 'mydomain.org',
 'HTTP_X_FORWARDED_PORT': '443',
 'HTTP_X_FORWARDED_PROTO': 'https',
 'HTTP_X_FORWARDED_SERVER': '46dc9412c09b',
 'HTTP_X_REAL_IP': '192.168.1.14',
 'HTTP_X_SCRIPT_NAME': '/my/prefix',
 'PATH_INFO': '/myuser/',
 'QUERY_STRING': '',
 'REMOTE_ADDR': '172.20.0.4',
 'REMOTE_HOST': '',
 'REQUEST_METHOD': 'PROPFIND',
 'SCRIPT_NAME': '',
 'SERVER_NAME': '5e562debb416',
 'SERVER_PORT': '5232',
 'SERVER_PROTOCOL': 'HTTP/1.0',
 'SERVER_SOFTWARE': 'WSGIServer/0.2',
 'wsgi.errors': <_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>,
 'wsgi.file_wrapper': <class 'wsgiref.util.FileWrapper'>,
 'wsgi.input': <_io.BufferedReader name=6>,
 'wsgi.multiprocess': False,
 'wsgi.multithread': True,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'http',
 'wsgi.version': (1, 0)}
[2022-09-04 20:34:15 +0000] [1/Thread-63 (process_request_thread)] [DEBUG] Base prefix (from HTTP_X_SCRIPT_NAME): '/my/prefix'
[2022-09-04 20:34:15 +0000] [1/Thread-64 (process_request_thread)] [INFO] PROPFIND request for '/myuser/' with depth '0' received from 172.20.0.4 (forwarded for '192.168.1.14, 172.20.0.2') using 'DAVx5/4.2.3.1-ose (2022/08/27; dav4jvm; okhttp/4.10.0) Android/12'
[2022-09-04 20:34:15 +0000] [1/Thread-63 (process_request_thread)] [DEBUG] Sanitized path: '/myuser/'
[2022-09-04 20:34:15 +0000] [1/Thread-64 (process_request_thread)] [DEBUG] Request headers:
{'CONTENT_LENGTH': '290',
 'CONTENT_TYPE': 'application/xml; charset=utf-8',
 'GATEWAY_INTERFACE': 'CGI/1.1',
 'HTTP_ACCEPT_ENCODING': 'gzip',
 'HTTP_ACCEPT_LANGUAGE': 'de-DE, de;q=0.7, *;q=0.5',
 'HTTP_AUTHORIZATION': 'Basic **masked**',
 'HTTP_CONNECTION': 'close',
 'HTTP_DEPTH': '0',
 'HTTP_HOST': 'webcal',
 'HTTP_USER_AGENT': 'DAVx5/4.2.3.1-ose (2022/08/27; dav4jvm; okhttp/4.10.0) '
                    'Android/12',
 'HTTP_X_FORWARDED_FOR': '192.168.1.14, 172.20.0.2',
 'HTTP_X_FORWARDED_HOST': 'mydomain.org',
 'HTTP_X_FORWARDED_PORT': '443',
 'HTTP_X_FORWARDED_PROTO': 'https',
 'HTTP_X_FORWARDED_SERVER': '46dc9412c09b',
 'HTTP_X_REAL_IP': '192.168.1.14',
 'HTTP_X_SCRIPT_NAME': '/my/prefix',
 'PATH_INFO': '/myuser/',
 'QUERY_STRING': '',
 'REMOTE_ADDR': '172.20.0.4',
 'REMOTE_HOST': '',
 'REQUEST_METHOD': 'PROPFIND',
 'SCRIPT_NAME': '',
 'SERVER_NAME': '5e562debb416',
 'SERVER_PORT': '5232',
 'SERVER_PROTOCOL': 'HTTP/1.0',
 'SERVER_SOFTWARE': 'WSGIServer/0.2',
 'wsgi.errors': <_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>,
 'wsgi.file_wrapper': <class 'wsgiref.util.FileWrapper'>,
 'wsgi.input': <_io.BufferedReader name=9>,
 'wsgi.multiprocess': False,
 'wsgi.multithread': True,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'http',
 'wsgi.version': (1, 0)}
[2022-09-04 20:34:15 +0000] [1/Thread-64 (process_request_thread)] [DEBUG] Base prefix (from HTTP_X_SCRIPT_NAME): '/my/prefix'
[2022-09-04 20:34:15 +0000] [1/Thread-64 (process_request_thread)] [DEBUG] Sanitized path: '/myuser/'
[2022-09-04 20:34:15 +0000] [1/Thread-63 (process_request_thread)] [INFO] Successful login: 'myuser'
[2022-09-04 20:34:15 +0000] [1/Thread-64 (process_request_thread)] [INFO] Successful login: 'myuser'
[2022-09-04 20:34:15 +0000] [1/Thread-63 (process_request_thread)] [DEBUG] Request content:
<?xml version="1.0"?>
<propfind xmlns="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav" xmlns:CS="http://calendarserver.org/ns/">
  <prop>
    <C:max-resource-size />
    <CS:getctag />
    <sync-token />
  </prop>
</propfind>

[2022-09-04 20:34:15 +0000] [1/Thread-64 (process_request_thread)] [DEBUG] Request content:
<?xml version="1.0"?>
<propfind xmlns="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav" xmlns:CS="http://calendarserver.org/ns/">
  <prop>
    <C:max-resource-size />
    <supported-report-set />
    <CS:getctag />
    <sync-token />
  </prop>
</propfind>

[2022-09-04 20:34:15 +0000] [1/Thread-63 (process_request_thread)] [INFO] Access to '/myuser/' denied for 'myuser'
[2022-09-04 20:34:15 +0000] [1/Thread-63 (process_request_thread)] [DEBUG] Response content:
Access to the requested resource forbidden.
[2022-09-04 20:34:15 +0000] [1/Thread-63 (process_request_thread)] [INFO] PROPFIND response status for '/myuser/' with depth '0' in 0.049 seconds: 403 Forbidden
[2022-09-04 20:34:15 +0000] [1/Thread-64 (process_request_thread)] [INFO] Access to '/myuser/' denied for 'myuser'
[2022-09-04 20:34:15 +0000] [1/Thread-64 (process_request_thread)] [DEBUG] Response content:
Access to the requested resource forbidden.

nginx config (TLS is done elsewhere):

events {
  worker_connections  1024;  ## Default: 1024
}
http{

upstream webcal {
  server radicale:5232;
}


server {
  listen 0.0.0.0:80;
  listen [::]:80 default_server;
  server_tokens off;

  location / {
    root /usr/share/nginx/html;
  }

  location /my/prefix/ {
    proxy_pass http://webcal/;
    proxy_set_header     X-Script-Name /my/prefix;
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass_header Authorization;
  }
}
}

Same works fine with 2.1.12 (except for some radicale.conf reordering)

copyrights avatar Sep 04 '22 21:09 copyrights

Have you tried running radicale --verify-storage? I had to remove .Radicale.props in my user's collection root due to [ERROR] Invalid collection 'username': 'VCALENDAR' must not have child collections

hberntsen avatar Jun 09 '23 19:06 hberntsen

When I run radicale -C /radicale.conf --verify-storage it delivers me a lot of this error messages.

[2023-06-18 10:20:45 +0000] [1] [DEBUG] Verifying collection 'someuser/caldav'
[2023-06-18 10:20:45 +0000] [1] [DEBUG] Verifying collection 'someuser/carddav'
[2023-06-18 10:20:45 +0000] [1] [ERROR] Invalid item '20120287-5D0EBC45-F5355A64.vcf' in 'someuser/carddav': Failed to load item '20120287-5D0EBC45-F5355A64.vcf' in 'someuser/carddav': Failed to serialize item None from 'someuser/carddav': 'VCARD components must contain at least 1 FN'
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/radicale/item/__init__.py", line 413, in serialize
    self._text = self.vobject_item.serialize()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/vobject/base.py", line 254, in serialize
    return behavior.serialize(self, buf, lineLength, validate)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/vobject/behavior.py", line 157, in serialize
    cls.validate(obj, raiseException=True)
  File "/usr/lib/python3.11/site-packages/vobject/behavior.py", line 93, in validate
    raise base.ValidateError(m .format(cls.name, val[0], key))
vobject.base.ValidateError: 'VCARD components must contain at least 1 FN'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/radicale/storage/multifilesystem/get.py", line 101, in _get
    cache_content = self._store_item_cache(
                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/radicale/storage/multifilesystem/cache.py", line 86, in _store_item_cache
    content = self._item_cache_content(item)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/radicale/storage/multifilesystem/cache.py", line 76, in _item_cache_content
    return CacheContent(item.uid, item.etag, item.serialize(), item.name,
                                  ^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/radicale/item/__init__.py", line 435, in etag
    self._etag = get_etag(self.serialize())
                          ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/radicale/item/__init__.py", line 415, in serialize
    raise RuntimeError("Failed to serialize item %r from %r: %s" %
RuntimeError: Failed to serialize item None from 'someuser/carddav': 'VCARD components must contain at least 1 FN'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/radicale/storage/multifilesystem/verify.py", line 37, in exception_cm
    yield
  File "/usr/lib/python3.11/site-packages/radicale/storage/multifilesystem/discover.py", line 87, in discover
    item = collection._get(href)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/radicale/storage/multifilesystem/get.py", line 104, in _get
    raise RuntimeError("Failed to load item %r in %r: %s" %
RuntimeError: Failed to load item '20120287-5D0EBC45-F5355A64.vcf' in 'someuser/carddav': Failed to serialize item None from 'someuser/carddav': 'VCARD components must contain at least 1 FN'

with 20120287-5D0EBC45-F5355A64.vcf being

BEGIN:VCARD
VERSION:3.0
N:SomeName;SomeThing;;;
TEL;type=HOME,VOICE:0123 4567
PRODID:-//ContactsKit //intsig.com //1.0
UID:20120287-5D0EBC45-F5355A64
X-RADICALE-NAME:20120287-5D0EBC45-F5355A64.vcf
END:VCARD%

copyrights avatar Jun 18 '23 10:06 copyrights

Carefully read the error messages....

vobject.base.ValidateError: 'VCARD components must contain at least 1 FN'

-> looks like the VCARD file is not that valid that "vobject" will accept it.

pbiering avatar Jun 18 '23 12:06 pbiering

@pbiering right, the question is why, because this data have been in radicale since I started using it years ago.

Anyhow, I wrote this little script to fix the missing FN.

from pathlib import Path

p = Path("/your/path/collections/collection-root/someuser/carddav")

for vcf in p.glob('*.vcf'):
    update = True
    newfn = ""
    uid = ""
    with open(vcf,"r", encoding='utf-8') as fp:
        lines = fp.readlines()
        for line in lines:
            if line.startswith("FN:"):
                update = False
                break
            elif line.startswith("N:"):
                newfn = " ".join([x for x in line.strip()[2:].split(';') if len(x)>0])
            elif line.startswith("UID:"):
                uid = line[3:].strip()
    if update:
        if len(newfn) <1:
            newfn = uid
        with open(vcf,"w") as fp:
            for line in lines:
                fp.write(line)
                if line.startswith("UID:"):
                    fp.write("FN:{}\n".format(newfn))

and at least radicale -C /radicale.conf --verify-storage drops no errors anymore

copyrights avatar Jun 18 '23 19:06 copyrights

Okay, I found a way to fix this (the cause is still unknown).

I had a user without caldav or carddav folder. In previous versions this was OK. The user is only used for caldav. My fix was to create a folder caldav and move all *.ics and the .Radicale.props into it.

btw, the script above fixes an issue I hadn't noticed yet.

copyrights avatar Jun 18 '23 21:06 copyrights