helm caldav/carddav: `principal ... does not provide caldav service` followed by 405 error

Describe your Issue

I can't seem to get my calendar and contacts synced on Android (but I also tried on my m1 macbook running macOS 13.4.1). This has never worked for me, but only now am I looking into it. The logs seem to suggest it starts to work with:

2023-07-11 15:01:53 105 [servicedetection.DavResourceFinder] Found current-user-principal: https://cloud.example.com/remote.php/dav/principals/users/jessebot/
2023-07-11 15:01:53 105 [at.bitfire.dav4jvm.BasicDigestAuthHandler] Adding Basic authorization header for https://cloud.example.com/remote.php/dav/principals/users/jessebot/
2023-07-11 15:01:53 105 [network.HttpClient] <-- 204 https://cloud.example.com/remote.php/dav/principals/users/jessebot/ (37ms)
2023-07-11 15:01:53 105 [network.HttpClient] date: Tue, 11 Jul 2023 13:01:53 GMT
2023-07-11 15:01:53 105 [network.HttpClient] strict-transport-security: max-age=15724800; includeSubDomains
2023-07-11 15:01:53 105 [network.HttpClient] access-control-allow-origin: *
2023-07-11 15:01:53 105 [network.HttpClient] access-control-allow-credentials: true
2023-07-11 15:01:53 105 [network.HttpClient] access-control-allow-methods: GET, PUT, POST, DELETE, PATCH, OPTIONS
2023-07-11 15:01:53 105 [network.HttpClient] access-control-allow-headers: X-Forwarded-For
2023-07-11 15:01:53 105 [network.HttpClient] access-control-max-age: 1728000
2023-07-11 15:01:53 105 [network.HttpClient] content-length: 0
2023-07-11 15:01:53 105 [network.HttpClient] <-- END HTTP (0-byte body)

But then I see this:

Principal https://cloud.example.com/remote.php/dav/principals/users/jessebot/ doesn't provide caldav service

folllowed by it seemingly trying the base domain again and returning this:

<head><title>405 Not Allowed</title></head>
<body>
<center><h1>405 Not Allowed</h1></center>
<hr><center>nginx/1.25.1</center>
</body>
</html>

(full verbosity logs in the next section)

I know there's kind of a lot of caldav/carddav forum posts already, but none of them cover using an nginx container and the ingress-nginx controller on k8s. The closest we have is a few issues referencing docker, but my current configs are fairly, if not fully, aligned with what seem to be all the docs across the nextcloud/server, nextcloud/docker, and nextcloud/helm repos. The other issue is that many of the forum posts are ancient referencing nextcloud versions 13-17ish.

Logs and Errors

So this log is actually from the DAVx5 app after following the guide in the server docs here (note: I've tried both with my personal username and password and also an app password I generated from my security settings and both fail with ...doesn't provide caldav service):

DAVx5 verbose logging

2023-07-11 15:01:53 105 [network.HttpClient] <-- END HTTP (419-byte, 231-gzipped-byte body)
2023-07-11 15:01:53 105 [servicedetection.DavResourceFinder] Found current-user-principal: https://cloud.example.com/remote.php/dav/principals/users/jessebot/
2023-07-11 15:01:53 105 [network.HttpClient] --> OPTIONS https://cloud.example.com/remote.php/dav/principals/users/jessebot/ h2
2023-07-11 15:01:53 105 [network.HttpClient] Content-Length: 0
2023-07-11 15:01:53 105 [network.HttpClient] Accept-Encoding: identity
2023-07-11 15:01:53 105 [network.HttpClient] User-Agent: DAVx5/4.3.4.1-ose (2023/06/16; dav4jvm; okhttp/4.11.0) Android/13
2023-07-11 15:01:53 105 [network.HttpClient] Accept-Language: en-NL, en;q=0.7, *;q=0.5
2023-07-11 15:01:53 105 [network.HttpClient] Host: cloud.example.com
2023-07-11 15:01:53 105 [network.HttpClient] Connection: Keep-Alive
2023-07-11 15:01:53 105 [network.HttpClient] Cookie: oc_sessionPassphrase=mvZ%2FR8ZizTqiwfl3Eb%2FSUgwUWL9H3FXLLXxsFyEWfWY0kzrPcknYY9lyJZdBUN1wPVqccKNzamEv0B5FhseZtMxQnrJqpUw%2BEvqcKj%2FslgEVPaNSD9uLcd2UlYtcQx5K; __Host-nc_sameSiteCookielax=true; __Host-nc_sameSiteCookiestrict=true; ocprwvfxs7k4=16ba82cdc7e9dddafecddebf347f39a8
2023-07-11 15:01:53 105 [network.HttpClient] --> END OPTIONS
2023-07-11 15:01:53 105 [at.bitfire.dav4jvm.BasicDigestAuthHandler] Adding Basic authorization header for https://cloud.example.com/remote.php/dav/principals/users/jessebot/
2023-07-11 15:01:53 105 [network.HttpClient] <-- 204 https://cloud.example.com/remote.php/dav/principals/users/jessebot/ (37ms)
2023-07-11 15:01:53 105 [network.HttpClient] date: Tue, 11 Jul 2023 13:01:53 GMT
2023-07-11 15:01:53 105 [network.HttpClient] strict-transport-security: max-age=15724800; includeSubDomains
2023-07-11 15:01:53 105 [network.HttpClient] access-control-allow-origin: *
2023-07-11 15:01:53 105 [network.HttpClient] access-control-allow-credentials: true
2023-07-11 15:01:53 105 [network.HttpClient] access-control-allow-methods: GET, PUT, POST, DELETE, PATCH, OPTIONS
2023-07-11 15:01:53 105 [network.HttpClient] access-control-allow-headers: X-Forwarded-For
2023-07-11 15:01:53 105 [network.HttpClient] access-control-max-age: 1728000
2023-07-11 15:01:53 105 [network.HttpClient] content-length: 0
2023-07-11 15:01:53 105 [network.HttpClient] <-- END HTTP (0-byte body)
2023-07-11 15:01:53 105 [servicedetection.DavResourceFinder] Principal https://cloud.example.com/remote.php/dav/principals/users/jessebot/ doesn't provide caldav service
2023-07-11 15:01:53 105 [servicedetection.DavResourceFinder] No principal found at user-given URL, trying to discover for domain cloud.example.com
2023-07-11 15:01:53 105 [servicedetection.DavResourceFinder] Looking up SRV records for _caldavs._tcp.cloud.example.com
2023-07-11 15:01:53 105 [util.DavUtils] Using Android 10+ DnsResolver
2023-07-11 15:01:53 105 [servicedetection.DavResourceFinder] Didn't find caldav service, trying at https://cloud.example.com:443
2023-07-11 15:01:53 105 [util.DavUtils] Using Android 10+ DnsResolver
2023-07-11 15:01:53 105 [servicedetection.DavResourceFinder] Trying to determine principal from initial context path=https://cloud.example.com/.well-known/caldav
2023-07-11 15:01:53 105 [network.HttpClient] --> PROPFIND https://cloud.example.com/.well-known/caldav h2
2023-07-11 15:01:53 105 [network.HttpClient] Depth: 0
2023-07-11 15:01:53 105 [network.HttpClient] User-Agent: DAVx5/4.3.4.1-ose (2023/06/16; dav4jvm; okhttp/4.11.0) Android/13
2023-07-11 15:01:53 105 [network.HttpClient] Accept-Language: en-NL, en;q=0.7, *;q=0.5
2023-07-11 15:01:53 105 [network.HttpClient] Content-Type: application/xml; charset=utf-8
2023-07-11 15:01:53 105 [network.HttpClient] Content-Length: 198
2023-07-11 15:01:53 105 [network.HttpClient] Host: cloud.example.com
2023-07-11 15:01:53 105 [network.HttpClient] Connection: Keep-Alive
2023-07-11 15:01:53 105 [network.HttpClient] Accept-Encoding: gzip
2023-07-11 15:01:53 105 [network.HttpClient] Cookie: oc_sessionPassphrase=mvZ%2FR8ZizTqiwfl3Eb%2FSUgwUWL9H3FXLLXxsFyEWfWY0kzrPcknYY9lyJZdBUN1wPVqccKNzamEv0B5FhseZtMxQnrJqpUw%2BEvqcKj%2FslgEVPaNSD9uLcd2UlYtcQx5K; __Host-nc_sameSiteCookielax=true; __Host-nc_sameSiteCookiestrict=true; ocprwvfxs7k4=16ba82cdc7e9dddafecddebf347f39a8
2023-07-11 15:01:53 105 [network.HttpClient]
2023-07-11 15:01:53 105 [network.HttpClient] <?xml version='1.0' encoding='UTF-8' ?><propfind xmlns="DAV:" xmlns:CAL="urn:ietf:params:xml:ns:caldav" xmlns:CARD="urn:ietf:params:xml:ns:carddav"><prop><current-user-principal /></prop></propfind>
2023-07-11 15:01:53 105 [network.HttpClient] --> END PROPFIND (198-byte body)
2023-07-11 15:01:53 105 [at.bitfire.dav4jvm.BasicDigestAuthHandler] Adding Basic authorization header for https://cloud.example.com/.well-known/caldav
2023-07-11 15:01:53 105 [network.HttpClient] <-- 301 https://cloud.example.com/.well-known/caldav (39ms)
2023-07-11 15:01:53 105 [network.HttpClient] date: Tue, 11 Jul 2023 13:01:53 GMT
2023-07-11 15:01:53 105 [network.HttpClient] content-type: text/html
2023-07-11 15:01:53 105 [network.HttpClient] content-length: 162
2023-07-11 15:01:53 105 [network.HttpClient] location: https://cloud.example.com/remote.php/dav/
2023-07-11 15:01:53 105 [network.HttpClient]
2023-07-11 15:01:53 105 [network.HttpClient] <html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>

2023-07-11 15:01:53 105 [network.HttpClient] <-- END HTTP (162-byte body)
2023-07-11 15:01:53 105 [at.bitfire.dav4jvm.DavResource] Redirected, new location = https://cloud.example.com/remote.php/dav/
2023-07-11 15:01:53 105 [network.HttpClient] --> PROPFIND https://cloud.example.com/remote.php/dav/ h2
2023-07-11 15:01:53 105 [network.HttpClient] Depth: 0
2023-07-11 15:01:53 105 [network.HttpClient] User-Agent: DAVx5/4.3.4.1-ose (2023/06/16; dav4jvm; okhttp/4.11.0) Android/13
2023-07-11 15:01:53 105 [network.HttpClient] Accept-Language: en-NL, en;q=0.7, *;q=0.5
2023-07-11 15:01:53 105 [network.HttpClient] Content-Type: application/xml; charset=utf-8
2023-07-11 15:01:53 105 [network.HttpClient] Content-Length: 198
2023-07-11 15:01:53 105 [network.HttpClient] Host: cloud.example.com
2023-07-11 15:01:53 105 [network.HttpClient] Connection: Keep-Alive
2023-07-11 15:01:53 105 [network.HttpClient] Accept-Encoding: gzip
2023-07-11 15:01:53 105 [network.HttpClient] Cookie: oc_sessionPassphrase=mvZ%2FR8ZizTqiwfl3Eb%2FSUgwUWL9H3FXLLXxsFyEWfWY0kzrPcknYY9lyJZdBUN1wPVqccKNzamEv0B5FhseZtMxQnrJqpUw%2BEvqcKj%2FslgEVPaNSD9uLcd2UlYtcQx5K; __Host-nc_sameSiteCookielax=true; __Host-nc_sameSiteCookiestrict=true; ocprwvfxs7k4=16ba82cdc7e9dddafecddebf347f39a8
2023-07-11 15:01:53 105 [network.HttpClient]
2023-07-11 15:01:53 105 [network.HttpClient] <?xml version='1.0' encoding='UTF-8' ?><propfind xmlns="DAV:" xmlns:CAL="urn:ietf:params:xml:ns:caldav" xmlns:CARD="urn:ietf:params:xml:ns:carddav"><prop><current-user-principal /></prop></propfind>
2023-07-11 15:01:53 105 [network.HttpClient] --> END PROPFIND (198-byte body)
2023-07-11 15:01:53 105 [at.bitfire.dav4jvm.BasicDigestAuthHandler] Adding Basic authorization header for https://cloud.example.com/remote.php/dav/
2023-07-11 15:01:53 105 [network.HttpClient] <-- 207 https://cloud.example.com/remote.php/dav/ (54ms)
2023-07-11 15:01:53 105 [network.HttpClient] date: Tue, 11 Jul 2023 13:01:53 GMT
2023-07-11 15:01:53 105 [network.HttpClient] content-type: application/xml; charset=utf-8
2023-07-11 15:01:53 105 [network.HttpClient] expires: Thu, 19 Nov 1981 08:52:00 GMT
2023-07-11 15:01:53 105 [network.HttpClient] cache-control: no-store, no-cache, must-revalidate
2023-07-11 15:01:53 105 [network.HttpClient] pragma: no-cache
2023-07-11 15:01:53 105 [network.HttpClient] content-security-policy: default-src 'none';
2023-07-11 15:01:53 105 [network.HttpClient] vary: Brief,Prefer
2023-07-11 15:01:53 105 [network.HttpClient] dav: 1, 3, extended-mkcol, access-control, calendarserver-principal-property-search, nextcloud-checksum-update, nc-calendar-search, nc-enable-birthday-calendar
2023-07-11 15:01:53 105 [network.HttpClient] x-request-id: SRrZC3eQQhFwWGenI5JA
2023-07-11 15:01:53 105 [network.HttpClient] x-debug-token: SRrZC3eQQhFwWGenI5JA
2023-07-11 15:01:53 105 [network.HttpClient] content-encoding: gzip
2023-07-11 15:01:53 105 [network.HttpClient] referrer-policy: no-referrer
2023-07-11 15:01:53 105 [network.HttpClient] x-content-type-options: nosniff
2023-07-11 15:01:53 105 [network.HttpClient] x-download-options: noopen
2023-07-11 15:01:53 105 [network.HttpClient] x-frame-options: SAMEORIGIN
2023-07-11 15:01:53 105 [network.HttpClient] x-permitted-cross-domain-policies: none
2023-07-11 15:01:53 105 [network.HttpClient] x-robots-tag: noindex, nofollow
2023-07-11 15:01:53 105 [network.HttpClient] x-xss-protection: 1; mode=block
2023-07-11 15:01:53 105 [network.HttpClient] strict-transport-security: max-age=15724800; includeSubDomains
2023-07-11 15:01:53 105 [network.HttpClient] access-control-allow-origin: *
2023-07-11 15:01:53 105 [network.HttpClient] access-control-allow-credentials: true
2023-07-11 15:01:53 105 [network.HttpClient] access-control-allow-methods: GET, PUT, POST, DELETE, PATCH, OPTIONS
2023-07-11 15:01:53 105 [network.HttpClient] access-control-allow-headers: X-Forwarded-For
2023-07-11 15:01:53 105 [network.HttpClient] access-control-max-age: 1728000
2023-07-11 15:01:53 105 [network.HttpClient]
2023-07-11 15:01:53 105 [network.HttpClient] <?xml version="1.0"?>
<d:multistatus xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:oc="http://owncloud.org/ns" xmlns:nc="http://nextcloud.org/ns"><d:response><d:href>/remote.php/dav/</d:href><d:propstat><d:prop><d:current-user-principal><d:href>/remote.php/dav/principals/users/jessebot/</d:href></d:current-user-principal></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response></d:multistatus>

2023-07-11 15:01:53 105 [network.HttpClient] <-- END HTTP (419-byte, 231-gzipped-byte body)
2023-07-11 15:01:53 105 [servicedetection.DavResourceFinder] Found current-user-principal: https://cloud.example.com/remote.php/dav/principals/users/jessebot/
2023-07-11 15:01:53 105 [network.HttpClient] --> OPTIONS https://cloud.example.com/remote.php/dav/principals/users/jessebot/ h2
2023-07-11 15:01:53 105 [network.HttpClient] Content-Length: 0
2023-07-11 15:01:53 105 [network.HttpClient] Accept-Encoding: identity
2023-07-11 15:01:53 105 [network.HttpClient] User-Agent: DAVx5/4.3.4.1-ose (2023/06/16; dav4jvm; okhttp/4.11.0) Android/13
2023-07-11 15:01:53 105 [network.HttpClient] Accept-Language: en-NL, en;q=0.7, *;q=0.5
2023-07-11 15:01:53 105 [network.HttpClient] Host: cloud.example.com
2023-07-11 15:01:53 105 [network.HttpClient] Connection: Keep-Alive
2023-07-11 15:01:53 105 [network.HttpClient] Cookie: oc_sessionPassphrase=mvZ%2FR8ZizTqiwfl3Eb%2FSUgwUWL9H3FXLLXxsFyEWfWY0kzrPcknYY9lyJZdBUN1wPVqccKNzamEv0B5FhseZtMxQnrJqpUw%2BEvqcKj%2FslgEVPaNSD9uLcd2UlYtcQx5K; __Host-nc_sameSiteCookielax=true; __Host-nc_sameSiteCookiestrict=true; ocprwvfxs7k4=16ba82cdc7e9dddafecddebf347f39a8
2023-07-11 15:01:53 105 [network.HttpClient] --> END OPTIONS
2023-07-11 15:01:53 105 [at.bitfire.dav4jvm.BasicDigestAuthHandler] Adding Basic authorization header for https://cloud.example.com/remote.php/dav/principals/users/jessebot/
2023-07-11 15:01:54 105 [network.HttpClient] <-- 204 https://cloud.example.com/remote.php/dav/principals/users/jessebot/ (41ms)
2023-07-11 15:01:54 105 [network.HttpClient] date: Tue, 11 Jul 2023 13:01:53 GMT
2023-07-11 15:01:54 105 [network.HttpClient] strict-transport-security: max-age=15724800; includeSubDomains
2023-07-11 15:01:54 105 [network.HttpClient] access-control-allow-origin: *
2023-07-11 15:01:54 105 [network.HttpClient] access-control-allow-credentials: true
2023-07-11 15:01:54 105 [network.HttpClient] access-control-allow-methods: GET, PUT, POST, DELETE, PATCH, OPTIONS
2023-07-11 15:01:54 105 [network.HttpClient] access-control-allow-headers: X-Forwarded-For
2023-07-11 15:01:54 105 [network.HttpClient] access-control-max-age: 1728000
2023-07-11 15:01:54 105 [network.HttpClient] content-length: 0
2023-07-11 15:01:54 105 [network.HttpClient] <-- END HTTP (0-byte body)
2023-07-11 15:01:54 105 [servicedetection.DavResourceFinder] Principal https://cloud.example.com/remote.php/dav/principals/users/jessebot/ doesn't provide caldav service
2023-07-11 15:01:54 105 [servicedetection.DavResourceFinder] Trying to determine principal from initial context path=https://cloud.example.com/
2023-07-11 15:01:54 105 [network.HttpClient] --> PROPFIND https://cloud.example.com/ h2
2023-07-11 15:01:54 105 [network.HttpClient] Depth: 0
2023-07-11 15:01:54 105 [network.HttpClient] User-Agent: DAVx5/4.3.4.1-ose (2023/06/16; dav4jvm; okhttp/4.11.0) Android/13
2023-07-11 15:01:54 105 [network.HttpClient] Accept-Language: en-NL, en;q=0.7, *;q=0.5
2023-07-11 15:01:54 105 [network.HttpClient] Content-Type: application/xml; charset=utf-8
2023-07-11 15:01:54 105 [network.HttpClient] Content-Length: 198
2023-07-11 15:01:54 105 [network.HttpClient] Host: cloud.example.com
2023-07-11 15:01:54 105 [network.HttpClient] Connection: Keep-Alive
2023-07-11 15:01:54 105 [network.HttpClient] Accept-Encoding: gzip
2023-07-11 15:01:54 105 [network.HttpClient] Cookie: oc_sessionPassphrase=mvZ%2FR8ZizTqiwfl3Eb%2FSUgwUWL9H3FXLLXxsFyEWfWY0kzrPcknYY9lyJZdBUN1wPVqccKNzamEv0B5FhseZtMxQnrJqpUw%2BEvqcKj%2FslgEVPaNSD9uLcd2UlYtcQx5K; __Host-nc_sameSiteCookielax=true; __Host-nc_sameSiteCookiestrict=true; ocprwvfxs7k4=16ba82cdc7e9dddafecddebf347f39a8
2023-07-11 15:01:54 105 [network.HttpClient]
2023-07-11 15:01:54 105 [network.HttpClient] <?xml version='1.0' encoding='UTF-8' ?><propfind xmlns="DAV:" xmlns:CAL="urn:ietf:params:xml:ns:caldav" xmlns:CARD="urn:ietf:params:xml:ns:carddav"><prop><current-user-principal /></prop></propfind>
2023-07-11 15:01:54 105 [network.HttpClient] --> END PROPFIND (198-byte body)
2023-07-11 15:01:54 105 [at.bitfire.dav4jvm.BasicDigestAuthHandler] Adding Basic authorization header for https://cloud.example.com/
2023-07-11 15:01:54 105 [network.HttpClient] <-- 405 https://cloud.example.com/ (36ms)
2023-07-11 15:01:54 105 [network.HttpClient] date: Tue, 11 Jul 2023 13:01:53 GMT
2023-07-11 15:01:54 105 [network.HttpClient] content-type: text/html
2023-07-11 15:01:54 105 [network.HttpClient] content-length: 157
2023-07-11 15:01:54 105 [network.HttpClient] referrer-policy: no-referrer
2023-07-11 15:01:54 105 [network.HttpClient] x-content-type-options: nosniff
2023-07-11 15:01:54 105 [network.HttpClient] x-download-options: noopen
2023-07-11 15:01:54 105 [network.HttpClient] x-frame-options: SAMEORIGIN
2023-07-11 15:01:54 105 [network.HttpClient] x-permitted-cross-domain-policies: none
2023-07-11 15:01:54 105 [network.HttpClient] x-robots-tag: noindex, nofollow
2023-07-11 15:01:54 105 [network.HttpClient] x-xss-protection: 1; mode=block
2023-07-11 15:01:54 105 [network.HttpClient] strict-transport-security: max-age=15724800; includeSubDomains
2023-07-11 15:01:54 105 [network.HttpClient] access-control-allow-origin: *
2023-07-11 15:01:54 105 [network.HttpClient] access-control-allow-credentials: true
2023-07-11 15:01:54 105 [network.HttpClient] access-control-allow-methods: GET, PUT, POST, DELETE, PATCH, OPTIONS
2023-07-11 15:01:54 105 [network.HttpClient] access-control-allow-headers: X-Forwarded-For
2023-07-11 15:01:54 105 [network.HttpClient] access-control-max-age: 1728000
2023-07-11 15:01:54 105 [network.HttpClient]
2023-07-11 15:01:54 105 [network.HttpClient] <html>
<head><title>405 Not Allowed</title></head>
<body>
<center><h1>405 Not Allowed</h1></center>
<hr><center>nginx/1.25.1</center>
</body>
</html>

2023-07-11 15:01:54 105 [network.HttpClient] <-- END HTTP (157-byte body)
2023-07-11 15:01:54 105 [servicedetection.DavResourceFinder] No resource found
EXCEPTION at.bitfire.dav4jvm.exception.HttpException: HTTP 405
        at at.bitfire.dav4jvm.DavResource.checkStatus(DavResource.kt:3)
        at at.bitfire.dav4jvm.DavResource.checkStatus(DavResource.kt:1)
        at at.bitfire.dav4jvm.DavResource.processMultiStatus(DavResource.kt:2)
        at at.bitfire.dav4jvm.DavResource.propfind(DavResource.kt:76)
        at at.bitfire.davdroid.servicedetection.DavResourceFinder.getCurrentUserPrincipal(DavResourceFinder.kt:38)
        at at.bitfire.davdroid.servicedetection.DavResourceFinder.discoverPrincipalUrl(DavResourceFinder.kt:305)
        at at.bitfire.davdroid.servicedetection.DavResourceFinder.findInitialConfiguration(DavResourceFinder.kt:31)
        at at.bitfire.davdroid.servicedetection.DavResourceFinder.findInitialConfiguration(DavResourceFinder.kt:4)
        at at.bitfire.davdroid.ui.setup.DetectConfigurationFragment$DetectConfigurationModel$detectConfiguration$2.invoke(DetectConfigurationFragment.kt:7)
        at at.bitfire.davdroid.ui.setup.DetectConfigurationFragment$DetectConfigurationModel$detectConfiguration$2.invoke(DetectConfigurationFragment.kt:1)
        at kotlin.concurrent.ThreadsKt$thread$thread$1.run(Thread.kt:3)

2023-07-11 15:02:00 2 [ui.DebugInfoActivity] Writing debug info to /data/user/0/at.bitfire.davdroid/files/debug/davx5-debug.zip

Describe your Environment

Kubernetes distribution: k3s
Helm Version (or App that manages helm): ArgoCD version 2.6.7
Helm Chart Version: 3.5.15 which uses the nextcloud 27.0.0 image

values.yaml:

image

image:
  repository: nextcloud
  flavor: fpm
  pullPolicy: Always
replicaCount: 1

ingress

ingress:
  enabled: true
  className: nginx
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 4G
    kubernetes.io/tls-acme: "true"
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-headers: "X-Forwarded-For"
    nginx.ingress.kubernetes.io/server-snippet: |-
      server_tokens off;
      proxy_hide_header X-Powered-By;

      rewrite ^/.well-known/webfinger /index.php/.well-known/webfinger last;
      rewrite ^/.well-known/nodeinfo /index.php/.well-known/nodeinfo last;
      rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
      rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json;

      location = /.well-known/carddav {
        return 301 $scheme://$host/remote.php/dav/;
      }

      location = /.well-known/caldav {
        return 301 $scheme://$host/remote.php/dav/;
      }

      location = /robots.txt {
        allow all;
        log_not_found off;
        access_log off;
      }

      location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ {
        proxy_buffer_size          128k;
        proxy_buffers              4 256k;
        proxy_busy_buffers_size    256k;
        deny all;
      }

      location ~ ^/(?:autotest|occ|issue|indie|db_|console) {
        proxy_buffer_size          128k;
        proxy_buffers              4 256k;
        proxy_busy_buffers_size    256k;
        deny all;
      }
  tls:
    - secretName: nextcloud-tls
      hosts:
        - cloud.example.com
  labels: {}
  path: /
  pathType: Prefix

nginx

nginx:
  ## You need to set an fpm version of the image for nextcloud if you want to use nginx!
  enabled: true
  image:
    repository: nginx
    tag: alpine
    pullPolicy: IfNotPresent

  config:
    # This generates the default nginx config as per the nextcloud documentation
    default: false
    custom: |-
      worker_processes auto;

      error_log  /var/log/nginx/error.log warn;
      pid        /var/run/nginx.pid;

      events {
        worker_connections  1024;
      }

      http {
        include       /etc/nginx/mime.types;
        default_type  application/octet-stream;

        log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"';

        access_log  /var/log/nginx/access.log  main;

        sendfile        on;
        #tcp_nopush     on;

        keepalive_timeout  65;

        #gzip  on;

        upstream php-handler {
            server 127.0.0.1:9000;
            # unsure if this still works, worked in php7.x-fpm
            server unix:/var/run/php/php8.1-fpm.sock;
        }

        server {
            listen 80;

            # set max upload size
            client_max_body_size 10G;
            fastcgi_buffers 64 4K;

            # Enable gzip but do not remove ETag headers
            gzip on;
            gzip_vary on;
            gzip_comp_level 4;
            gzip_min_length 256;
            gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
            gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;

            # Pagespeed is not supported by Nextcloud, so if your server is built
            # with the `ngx_pagespeed` module, uncomment this line to disable it.
            #pagespeed off;

            # HTTP response headers borrowed from Nextcloud `.htaccess`
            add_header Referrer-Policy                      "no-referrer"       always;
            add_header X-Content-Type-Options               "nosniff"           always;
            add_header X-Download-Options                   "noopen"            always;
            add_header X-Frame-Options                      "SAMEORIGIN"        always;
            add_header X-Permitted-Cross-Domain-Policies    "none"              always;
            add_header X-Robots-Tag                         "noindex, nofollow" always;
            add_header X-XSS-Protection                     "1; mode=block"     always;

            # Remove X-Powered-By, which is an information leak
            fastcgi_hide_header X-Powered-By;

            # Path to the root of your installation
            root /var/www/html;

            # Specify how to handle directories -- specifying `/index.php$request_uri`
            # here as the fallback means that Nginx always exhibits the desired behaviour
            # when a client requests a path that corresponds to a directory that exists
            # on the server. In particular, if that directory contains an index.php file,
            # that file is correctly served; if it doesn't, then the request is passed to
            # the front-end controller. This consistent behaviour means that we don't need
            # to specify custom rules for certain paths (e.g. images and other assets,
            # `/updater`, `/ocm-provider`, `/ocs-provider`), and thus
            # `try_files $uri $uri/ /index.php$request_uri`
            # always provides the desired behaviour.
            index index.php index.html /index.php$request_uri;

            # Rule borrowed from `.htaccess` to handle Microsoft DAV clients
            location = / {
                if ( $http_user_agent ~ ^DavClnt ) {
                    return 302 /remote.php/webdav/$is_args$args;
                }
            }

            location = /robots.txt {
                allow all;
                log_not_found off;
                access_log off;
            }

            # Make a regex exception for `/.well-known` so that clients can still
            # access it despite the existence of the regex rule
            # `location ~ /(\.|autotest|...)` which would otherwise handle requests
            # for `/.well-known`.
            location ^~ /.well-known {
                # The following 6 rules are borrowed from `.htaccess`

                location = /.well-known/carddav     { return 301 /remote.php/dav/; }
                location = /.well-known/caldav      { return 301 /remote.php/dav/; }
                # Anything else is dynamically handled by Nextcloud
                location ^~ /.well-known            { return 301 /index.php$uri; }

                try_files $uri $uri/ =404;
            }

            # Rules borrowed from `.htaccess` to hide certain paths from clients
            location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)(?:$|/)  { return 404; }
            location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console)              { return 404; }

            # Ensure this block, which passes PHP files to the PHP process, is above the blocks
            # which handle static assets (as seen below). If this block is not declared first,
            # then Nginx will encounter an infinite rewriting loop when it prepends `/index.php`
            # to the URI, resulting in a HTTP 500 error response.
            location ~ \.php(?:$|/) {
                # Required for legacy support
                rewrite ^/(?!index|remote|public|cron|core\/ajax\/update|status|ocs\/v[12]|updater\/.+|oc[ms]-provider\/.+|.+\/richdocumentscode\/proxy) /index.php$request_uri;

                fastcgi_split_path_info ^(.+?\.php)(/.*)$;
                set $path_info $fastcgi_path_info;

                try_files $fastcgi_script_name =404;

                include fastcgi_params;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                fastcgi_param PATH_INFO $path_info;
                #fastcgi_param HTTPS on;

                # Avoid sending the security headers twice
                fastcgi_param modHeadersAvailable true;
                fastcgi_param front_controller_active true; # Enable pretty urls
                fastcgi_pass php-handler;

                fastcgi_intercept_errors on;
                fastcgi_request_buffering off;
            }

            location ~ \.(?:css|js|svg|gif)$ {
                try_files $uri /index.php$request_uri;
                expires 6M;         # Cache-Control policy borrowed from `.htaccess`
                access_log off;     # Optional: Don't log access to assets
            }

            location ~ \.woff2?$ {
                try_files $uri /index.php$request_uri;
                expires 7d;         # Cache-Control policy borrowed from `.htaccess`
                access_log off;     # Optional: Don't log access to assets
            }

            location / {
                try_files $uri $uri/ /index.php$request_uri;
            }
        }
      }

  resources: {}

Additional context, if any

I have nginx enabled with this helm chart.
I use TOTP, but have also tried to use an app password with DAVx5, which unfortunatley results in the same issue :(
this is the only warning in nextcloud, but I don't think it's related:

screenshot of admin overview page

there is nothing in logs page of the nextcloud admin panel when I test this and refresh
I also tried to download a mobile profile config from the "Mobile & Desktop" page in personal settings here:

screenshot of Mobile & Desktop setting

but it also throw errors when the profile is imported:

macOS error example

I also checked the docs for sabre, but didn't see anything of value :(

Jul 11 '23 13:07 jessebot

I use nginx and ingress-nginx and calendar and contacts sync work for me :thinking:. We can try to compare configs and see why it doesn't work for you.

Jul 11 '23 13:07 provokateurin

Yes please, Thank you! 🙏 All of my configs are above under the "values.yaml" section. The ingress annotations seem to match the ones we have in this repo directly. I only had a slightly different nginx.config for a while before a kind soul merged a fix into this repo a little bit ago, but now I think it's the same too.

I also went and confirmed that I don't have modsecurity on yet (I'm sure that will be it's own can of worms when I turn that on 😂 ).

Jul 11 '23 14:07 jessebot

This is my whole values.yaml

image:
  flavor: fpm-alpine
nextcloud:
  host: [REDACTED]
ingress:
  enabled: true
  className: nginx
  tls:
    - hosts:
        - [REDACTED]
      secretName: [REDACTED]
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 10G
    nginx.ingress.kubernetes.io/proxy-connect-timeout: 10m
    nginx.ingress.kubernetes.io/proxy-read-timeout: 10m
    nginx.ingress.kubernetes.io/proxy-send-timeout: 10m
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
    cert-manager.io/cluster-issuer: letsencrypt-prod
    external-dns.alpha.kubernetes.io/ttl: 1m
    external-dns.alpha.kubernetes.io/target: [REDACTED]
nginx:
  enabled: true
internalDatabase:
  enabled: false
externalDatabase:
  enabled: true
  type: postgresql
  host: "acid-nextcloud-cluster:5432"
  existingSecret:
    enabled: true
    secretName: nextcloud.acid-nextcloud-cluster.credentials.postgresql.acid.zalan.do
    usernameKey: username
    passwordKey: password
redis:
  enabled: true
persistence:
  enabled: true
  existingClaim: nextcloud
rbac:
  enabled: true
cronjob:
  enabled: true

with chart version 3.5.14. I use https://github.com/kubernetes/ingress-nginx 4.5.2 right now.

Jul 11 '23 14:07 provokateurin

I guess the best way to test is to disable all your nginx customizations

Jul 11 '23 14:07 provokateurin

Awesome, thank you for all the details!

I'm also running the ingress-nginx-4.7.1 helm chart.

I agree, so I went ahead and removed most of the stuff I had there, and now my the relevant sections of my image, ingress, and nginx sections of my values.yaml looks like this:

image:
  flavor: fpm

ingress:
  enabled: true
  className: nginx
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 10G
    kubernetes.io/tls-acme: "true"
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-headers: "X-Forwarded-For"
  tls:
    - secretName: nextcloud-tls
      hosts:
        - cloud.example.com
  
nginx:
  enabled: true

# not sure if it matters, but I also have this trusted_proxies thing here:
nextcloud:
  configs:
    # adding your local ip might help on a self-hosted instance on your home network
    proxy.config.php: |-
      <?php
      $CONFIG = array (
        'trusted_proxies' => array(
          0 => '127.0.0.1',
          1 => '10.0.0.0/8'
        ),
        'forwarded_for_headers' => array('HTTP_X_FORWARDED_FOR'),
      );

Pretty grateful to have removed a bunch of stuff that wasn't necessary, because most everything is still working, and this simplifies my troubleshooting a lot, but the same errors persist when I try to use the android nextcloud app with DAVx5 🤔 The macOS contacts/calendar apps stopped giving auth errors, but also aren't syncing anything. No logs in the admin panel except this:

[PHP] Error: Optional parameter $trustedServers declared before required parameter $groupManager is implicitly treated as a required parameter at /var/www/html/apps/dav/lib/CardDAV/SystemAddressbook.php#60

PROPFIND /remote.php/dav/addressbooks/users/jessebot/
from REDACTED_IP_ADDR by jessebot at 2023-07-11T16:12:24+00:00

I think that's just from when I tried to import my vcards earlier, which is because of https://github.com/nextcloud/server/issues/38772 which should be solved soonish as there's an RC PR here, https://github.com/nextcloud/server/pull/39282, so I assume we'll get a new docker tag soonish. I don't know if that's partially breaking things though.

Oh also after removing all the nginx configs and ingress annotations I started getting this in the admin overview 🤔 :

Jul 11 '23 16:07 jessebot

I also see those warnings about well-know, but never bothered to investigate. As far as I know that should work ootb?

Jul 11 '23 16:07 provokateurin

I also see those warnings about well-know, but never bothered to investigate.

Ah yeah, I only recently got some time off work to start looking into them myself :) Added back the following ingress.annotations and that resolved the admin panel errors about "/.well-known/*":

    nginx.ingress.kubernetes.io/server-snippet: |-
      proxy_hide_header X-Powered-By;
      rewrite ^/.well-known/webfinger /index.php/.well-known/webfinger last;
      rewrite ^/.well-known/nodeinfo /index.php/.well-known/nodeinfo last;
      rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
      rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json;

      location = /.well-known/carddav {
        return 301 $scheme://$host/remote.php/dav/;
      }

      location = /.well-known/caldav {
        return 301 $scheme://$host/remote.php/dav/;
      }

but caldav/carddav still throws the same errors ;(

_{✏️ edited to remove the robots.txt snippet that wasn't necessary}

Jul 11 '23 16:07 jessebot

Have you enabled debug logging on the server to see if there is any more info in the logs?

Jul 11 '23 16:07 provokateurin

I plan on coming back to this after v27.0.1 of Nextcloud is released and available via docker, because it very may well be related to the bug I mentioned, and I haven't had a chance to enable debug logs (currently have it set to info), but I will revisit this soonish!

Looks like 27.0.1rc2 just dropped a couple of days ago, and for 27.0.0 they did 4 release candidates before settling, so probably another week or two and I'll upgrade, and approach this again with fresh eyes :) Thanks for all your help rubber-ducking in the meantime, Kate!

Update

The nextcloud server team did the release :D Here's the PR: https://github.com/nextcloud/helm/pull/419 After I figure out upgrades, I'll come back to this issue.

Jul 20 '23 08:07 jessebot

Hey. I have the same issue. My values on ingress are very similar to the original post. As soon as I set enable-cors to false on ingress (nginx.ingress.kubernetes.io/enable-cors: "false"), I managed to add my account for sync in official nextcloud app via DAVx5 and calendars and contacts seem to start synchronizing.

Jul 28 '23 04:07 qlonik

Thanks @qlonik for chiming in! enable-cors should work though 🤔 Let me test and get back to you since #419 has been merged now.

Jul 29 '23 13:07 jessebot

oh, I wonder if we can add something like the allow header like they suggest in this stack overflow post:

location / {
    if ($http_origin ~* "^https?://(nextclouddomain.com/remote.php/dav//|www.nextcloudomain.com/remote.php/dav/)$") {
        add_header Access-Control-Allow-Origin "$http_origin";
    }
}

I haven't had a chance to look into this due to a P1 with longhorn, but will update when I get back to testing this. If anyone in the meantime figures out the magic allow header to make this work either in the nginx.conf or a config-snippet in for ingress controller annotations, I'd love to see what you came up with :)

Jul 31 '23 13:07 jessebot

@provokateurin thanks for sharing what's working for you. Can you share your config for ingress-nginx? Are you using use-forwarded-headers?

I'm also running into this. I'm using Digital Ocean LB in front of ingress-nginx.

Nov 18 '23 18:11 adborden

I had to resetup DAVx5 on my phone and now I run into the same problem

Nov 29 '23 16:11 provokateurin

I'm going to take a look at this today.

I am pretty sure this got nothing to do with CORS because that is only interesting for browsers and any other clients don't care about it since it is not a server restriction.

My plan is to do a git bisect on this repo because I'm fairly sure that it is not a bug in a specific Nextcloud version since loads of other people outside of this helm chart would have the same problem.

Nov 30 '23 08:11 provokateurin

So I tried a lot of different things and I still have no clue what is going on. For some reason I end up with

<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>Sabre\DAV\Exception\NotAuthenticated</s:exception>
  <s:message>No public access to this resource., No 'Authorization: Basic' header found. Either the client didn't send one, or the server is misconfigured, No 'Authorization: Bearer' header found. Either the client didn't send one, or the server is mis-configured, No 'Authorization: Basic' header found. Either the client didn't send one, or the server is misconfigured</s:message>
</d:error>

when doing curl -v -X PROPFIND "http://cloud.example.com/.well-known/caldav" -L on my production instance. With the exact same setup locally (terraform+k3d) I can not reproduce this and I also don't see the errors in the admin overview. I don't understand how there can be any difference between the two since they are using the exact same terraform config (other than hostnames).

Now I was wondering if it might have to do with some state that is bad inside the instance since there seems to be no difference really.

Sadly I can't test if my local instance works with DAVx5 :sweat_smile:

Nov 30 '23 19:11 provokateurin

I tried my curl command against cloud.nextcloud.com and end up with the same unauthorized error. I am still able to successfully setup DAVx5 for that instance so it is probably not the source of the problems :/

Nov 30 '23 19:11 provokateurin

@Northcode since you worked on https://github.com/nextcloud/helm/pull/241, do you mind sharing your setup and if it works as expected?

Nov 30 '23 19:11 provokateurin

Thank you @provokateurin for taking the lead on this 💙

Dec 01 '23 08:12 jessebot

@jessebot @adborden @qlonik Could all of you try to setup a fresh instance with your setup if possible? Maybe you will have the same as me where a fresh instance works just fine.

Dec 01 '23 09:12 provokateurin

I just checked the admin overview again and saw this in the logs. The weird thing here is that it says http and not https since my prod instance is running on https. Looking at the redirect requests I also see that they only go to http and not https. I think this could be the problem here, although I don't understand why. I assume this might be because the nginx inside the container only does http and the ingress nginx does the https.

Dec 02 '23 10:12 provokateurin

Yes my assumption was correct! You need to uncomment the nginx.ingress.kubernetes.io/server-snippet in the ingress annotations. The ingress needs to handle the redirects because only it knows about https. I still see a complaint about webfinger not being properly setup (will investigate), but I got DAVx5 to work!

(The reason why I wasn't able to reproduce it locally with my setup is that it doesn't use https.)

Dec 02 '23 10:12 provokateurin

I can't find a difference between nodeinfo and webfinger so I'm not sure why that isn't working now :woman_shrugging:

Edit: The webfinger thing is a weird caching issue of my browser. In a new private window it works just fine.

Dec 02 '23 10:12 provokateurin

reopening till I can test too :)

Dec 04 '23 07:12 jessebot

Ah sorry, I thought you did test it already :see_no_evil:

Dec 04 '23 08:12 provokateurin

Btw I think we could remove the robots.txt and deny all parts from the server snippet. In theory it will reduce the load because we can already deny requests at the ingress controller, but in practice it doesn't change much and only makes the server snippet more complex. @jessebot do you agree we can remove it?

Dec 08 '23 10:12 provokateurin

I am pretty sure this got nothing to do with CORS because that is only interesting for browsers and any other clients don't care about it since it is not a server restriction.

@provokateurin but if you enable CORS in the nginx, doesn't that instruct nginx to reply to pre-flight requests, which use ... the OPTIONS method? https://developer.mozilla.org/en-US/docs/Glossary/Preflight_request

Jan 06 '24 21:01 pschichtel

https://github.com/nextcloud/helm/commit/67ea6eabcd9e429a1836f615ec8c93b8badc3c81 says this is needed for source-ip preservation, but I don't see how CORS is involved in that and it's definitely a routing conflict. I guess some server snippet could be added to circumvent this conflict, but I don't think it's helpful anyway

Jan 06 '24 21:01 pschichtel

I'm not sure what you mean. The problem here was that the ingress needed to do the redirect already because nextcloud itself (or it's nginx) doesn't know about https.

Jan 06 '24 21:01 provokateurin

I was facing this exact issue even though I had the redirect in-place on nginx' side. The wrong redirect does not manifest itself with the error mentioned in the title here, however the OPTIONS routings conflict caused by the cors option in ingress-nginx will produce this exact error, because the OPTIONS request does not contain the dav header expected by clients (since ingress-nginx handles the request without passing it to nextcloud thinking it's a CORS-preflight-request).

Jan 06 '24 21:01 pschichtel

helm helm copied to clipboard

caldav/carddav: `principal ... does not provide caldav service` followed by 405 error

Describe your Issue

Logs and Errors

Describe your Environment

values.yaml:

Additional context, if any

Update

helm
helm copied to clipboard