server icon indicating copy to clipboard operation
server copied to clipboard
verified publisher icon nextcloud

[Bug]: CalDAV free/busy request on user's `outbox` collection always returns `3.7;Could not find principal`

Open mcrha opened this issue 11 months ago • 10 comments

⚠️ This issue respects the following points: ⚠️

Bug description

Coming from downstream
https://gitlab.gnome.org/GNOME/evolution-data-server/-/issues/467

The free/busy requests over CalDAV always return 3.7;Could not find principal, even when asking for the logged in user

Steps to reproduce

Send a request to the server, to the user's scheduling Outbox, which is used for free/busy lookup:

BEGIN:VCALENDAR
PRODID:-//Mozilla.org/NONSGML Mozilla Calendar V1.1//EN
VERSION:2.0
METHOD:REQUEST
BEGIN:VFREEBUSY
UID:5328b732-5010-47b3-8bac-9bb36c2e51b8
DTSTAMP:20250117T114144Z
DTSTART:20250117T070000Z
DTEND:20250202T070000Z
ORGANIZER:mailto:[email protected]
ATTENDEE;PARTSTAT=NEEDS-ACTION;ROLE=REQ-PARTICIPANT;CUTYPE=INDIVIDUAL:mailt
 o:[email protected]
END:VFREEBUSY
END:VCALENDAR

The server responds:

<?xml version="1.0" encoding="utf-8"?>
<cal:schedule-response xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:cal="urn:ietf:params:xml:ns:caldav" xmlns:cs="http://calendarserver.org/ns/">
  <cal:response>
    <cal:recipient>
      <d:href>mailto:[email protected]</d:href>
    </cal:recipient>
    <cal:request-status>3.7;Could not find principal</cal:request-status>
  </cal:response>
</cal:schedule-response>

regardless the attendee is or is not a legitimate user configured on the server.

Expected behavior

The user is found, and its free/busy information is returned from the server.

It seems searchPrincipals() function is called with principals $prefixPath, which is not known to the function itself. When I add it beside the principals/users case, then it can find the user, but it fails later on. The change looks like:

--- nextcloud/apps/dav/lib/Connector/Sabre/Principal.php.orig	2019-05-15 10:39:32.000000000 -0400
+++ nextcloud/apps/dav/lib/Connector/Sabre/Principal.php	2025-01-17 09:51:10.089227728 -0500
@@ -331,6 +331,7 @@ class Principal implements BackendInterf
 		}
 
 		switch ($prefixPath) {
+			case 'principals':
 			case 'principals/users':
 				return $this->searchUserPrincipals($searchProperties, $test);

More details can be found in the downstream bug.

I do not know how to figure out the exact server version from the web UI, the "About" menu opens some marketing stuff, not a version info like in other apps, thus I cannot tell what precise version this is, I'm sorry, but looking into the latest code the above change is still applicable.

Nextcloud Server version

28

Operating system

None

PHP engine version

None

Web server

None

Database engine version

None

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

  • [ ] Default user-backend (database)
  • [ ] LDAP/ Active Directory
  • [ ] SSO - SAML
  • [ ] Other

Configuration report

List of activated Apps

Nextcloud Signing status

Nextcloud Logs

Additional info

No response

mcrha avatar Jan 17 '25 15:01 mcrha

cc @SebastianKrupinski @st3iny @tcitworld

kesselb avatar Jan 18 '25 18:01 kesselb

Hi, thanks for the report.

Does a user with the email [email protected] exist on the server? Is this email address the primary email address of the user? Please have a look a the personal settings of the user.

EDIT: Correct the email address.

st3iny avatar Feb 03 '25 14:02 st3iny

I changed the address for privacy reason in the description. It fails also for the user itself, the one being logged in. It's the only email address the user has set, thus might be the primary.

If you re-read the description, you'll see there is a block with a patch there - the function the patch touches never tries to search for the user, due to the parameter being different than expected. If I correct it (apply the patch), it fails later on, which was beyond my abilities to investigate it further.

Maybe for easier testing, the client runs this query over https::

POST /remote.php/dav/calendars/mcrha/outbox/ HTTP/1.1
Content-Type: text/calendar; charset="utf-8"

BEGIN:VCALENDAR
CALSCALE:GREGORIAN
PRODID:-//Ximian//NONSGML Evolution Calendar//EN
VERSION:2.0
METHOD:REQUEST
BEGIN:VFREEBUSY
UID:fa2566720b5589ed27115df0ec3b16dfaf098b61
DTSTAMP:20250203T164701Z
DTSTART:20250126T230000Z
DTEND:20250302T230000Z
ORGANIZER:mailto:mcrha
ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=CHAIR;PARTSTAT=NEEDS-ACTION;RSVP=FALSE:mailto:[email protected]
END:VFREEBUSY
END:VCALENDAR

and the server responds:

HTTP/1.1 403 Forbidden
Content-Type: application/xml; charset=utf-8
Server: nginx

<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
	<s:exception>Internal Server Error</s:exception>
	<s:message>
		The server was unable to complete your request.		If this happens again, please send the technical details below to the server administrator.		More details can be found in the server log.			</s:message>

	<s:technical-details>
		<s:remote-address>0.0.0.0</s:remote-address>
		<s:request-id>aGIOMI3gtGtnUmzi6iL3</s:request-id>

		</s:technical-details>

mcrha avatar Feb 03 '25 16:02 mcrha

Okay, I see the response is slightly different than in the description. It's what I get when I apply the patch. I re-tried with a newer Nexcloud (but I do not know where to find the exact version; if you tell me where it is in the web interface, then I can paste it here), which might have the part with the patch changed already or something, I cannot tell for sure, because I do not have access to the server itself. The line is 401 for the patch at the moment, inside the searchPrincipals() function.

I verified the logged-in user's email address is the one I used in the ATTENDEE property of the component, thus there's everything as it should be.

mcrha avatar Feb 03 '25 17:02 mcrha

Recently updated to Nextcloud 31 on an instance with ~600 users. Only some users' free/busy information could be fetched since update. It used to work on Nextcloud 30. I was able to fix the problem for us with the info on this issue (thanks to @mcrha ).

Most of our users have their email address in oc_preferences table stored with upper and lowercase chars. LDAP / Active Directory users certainly get their email address from the AD email field and it's like "[email protected]"

On the file lib/private/AllConfig.php, line 406, there is a function getUsersForUserValueCaseInsensitive which claims in his name to perform a case insensitive search, but it doesn't ! On line 412 (edit : my mistake, it's not exactly here, see my post further), the searchUsersByValueString function is called omitting the last argument, and this last argument is the one switching to a case sensitive or insensitive search. By default, it is set to false in the lib/private/Config/UserConfig.php file (line 409). Omitting the argument, the search is case sensitive, and all of our users having uppercase chars in their email address on the oc_preferences tables can't be fetched by the function, returning 3.7;Could not find principal

I don't know how to propose this modification to the community (I'm not familiar with github), so if someone could do this for me :

Line 412 of lib/private/AllConfig.php:

//$result = iterator_to_array(\OCP\Server::get(IUserConfig::class)->searchUsersByValueString($appName, $key, $value));
$result = iterator_to_array(\OCP\Server::get(IUserConfig::class)->searchUsersByValueString($appName, $key, $value, true));

I'm not sure this modification will fix the OP problem. But it's working for us.

Mer0me avatar Jun 04 '25 06:06 Mer0me

@Mer0me Thanks for the investigation. This already seems to be the case on the master and stable31 branches.

st3iny avatar Jun 04 '25 15:06 st3iny

@mcrha I have another question for you to debug this further: Could it be that there are multiple users on your system sharing the same email address?

You can find out quickly by running the following shell script:

echo 'select data from oc_accounts;' | mysql [...] | tail -n +2 | jq -r '.email.value | select(. != "") | select(. != null)' | sort | uniq -d

st3iny avatar Jun 04 '25 15:06 st3iny

@Mer0me Thanks for the investigation. This already seems to be the case on the master and stable31 branches.

My mistake. The omitted argument is in the getUsersForUserValue function, line 392 :

    public function getUsersForUserValue($appName, $key, $value) {
            /** @var list<string> $result */
           //$result = iterator_to_array(\OCP\Server::get(IUserConfig::class)->searchUsersByValueString($appName, $key, $value));
            $result = iterator_to_array(\OCP\Server::get(IUserConfig::class)->searchUsersByValueString($appName, $key, $value), true);
            return $result;
    }`

`

Mer0me avatar Jun 04 '25 15:06 Mer0me

That mysql [...] command for a locally running server precisely looks like what, please? It's a long time I played with the mysql database.

I do not think there are multiple users with the same email address, but I'll be happy to verify & confirm that.

mcrha avatar Jun 04 '25 16:06 mcrha

On v31.0.6, the problem mentioned on https://github.com/nextcloud/server/issues/50239#issuecomment-2940422659 is still present

Mer0me avatar Jun 13 '25 06:06 Mer0me

@Mer0me Email addresses should always be saved lowercase. The logic indeed breaks down if some email addresses are mixed case.

Would it be possible for you to try the following patch? It removes the casing check.

diff --git a/lib/private/AllConfig.php b/lib/private/AllConfig.php
index 72af6c960a5..ff12cff9be9 100644
--- a/lib/private/AllConfig.php
+++ b/lib/private/AllConfig.php
@@ -404,10 +404,6 @@ class AllConfig implements IConfig {
 	 * @deprecated 31.0.0 - use {@see IUserConfig::searchUsersByValueString} directly
 	 */
 	public function getUsersForUserValueCaseInsensitive($appName, $key, $value) {
-		if ($appName === 'settings' && $key === 'email') {
-			return $this->getUsersForUserValue($appName, $key, strtolower($value));
-		}
-
 		/** @var list<string> $result */
 		$result = iterator_to_array(\OCP\Server::get(IUserConfig::class)->searchUsersByValueString($appName, $key, $value, true));
 		return $result;

st3iny avatar Jun 19 '25 14:06 st3iny

Thanks, I'll try this patch tomorrow. If the email addresses are mixed case, it's certainly because they've been imported directly as they are from Active Directory with Nextcloud LDAP connector. If Nextcloud is designed to use only lowercase email addresses, maybe the LDAP connector should be checked too ?

Mer0me avatar Jun 19 '25 14:06 Mer0me

I also created a PR here: https://github.com/nextcloud/server/pull/53608

It seems like the email addresses are taken as is from LDAP which is not good. We still need a migration to fix all existing users though.

st3iny avatar Jun 19 '25 16:06 st3iny

@Mer0me Email addresses should always be saved lowercase. The logic indeed breaks down if some email addresses are mixed case.

Would it be possible for you to try the following patch? It removes the casing check.

diff --git a/lib/private/AllConfig.php b/lib/private/AllConfig.php index 72af6c960a5..ff12cff9be9 100644 --- a/lib/private/AllConfig.php +++ b/lib/private/AllConfig.php @@ -404,10 +404,6 @@ class AllConfig implements IConfig {

  • @deprecated 31.0.0 - use {@see IUserConfig::searchUsersByValueString} directly */ public function getUsersForUserValueCaseInsensitive($appName, $key, $value) {
  • if ($appName === 'settings' && $key === 'email') {

  • 	return $this->getUsersForUserValue($appName, $key, strtolower($value));

  • }

  • /** @var list<string> $result */
$result = iterator_to_array(\OCP\Server::get(IUserConfig::class)->searchUsersByValueString($appName, $key, $value, true));
return $result;

It works for us. Thanks.

Mer0me avatar Jun 20 '25 06:06 Mer0me

Thanks for testing. I'll inquire internally if we can accept this fix. I suspect it might cause negative performance regressions.

st3iny avatar Jun 20 '25 08:06 st3iny

@mcrha Please try the patch as well. Feedback would be greatly appreciated to see if you are facing the same issue with mixed case email addresses.

st3iny avatar Jun 20 '25 08:06 st3iny

My problem was not about case sensitivity, but about unhandled case (see the "patch" in the description)

mcrha avatar Jun 20 '25 08:06 mcrha

Thanks for confirming. So we are talking about two different issues here.

st3iny avatar Jun 20 '25 08:06 st3iny

I still can't reproduce the initial problem. I tried latest master, stable31 and stable30. I posted the given Free/Busy ICS file (with another set of email addresses though).

Reply looked like this:

<?xml version="1.0" encoding="utf-8"?>
<cal:schedule-response xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:cal="urn:ietf:params:xml:ns:caldav" xmlns:cs="http://calendarserver.org/ns/" xmlns:oc="http://owncloud.org/ns" xmlns:nc="http://nextcloud.org/ns">
  <cal:response>
    <cal:recipient>
      <d:href>mailto:[email protected]</d:href>
    </cal:recipient>
    <cal:request-status>2.0;Success</cal:request-status>
    <cal:calendar-data>BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Sabre//Sabre VObject 4.5.6//EN
CALSCALE:GREGORIAN
METHOD:REPLY
BEGIN:VFREEBUSY
DTSTART:20250117T070000Z
DTEND:20250202T070000Z
DTSTAMP:20250620T085035Z
ATTENDEE:mailto:[email protected]
UID:5328b732-5010-47b3-8bac-9bb36c2e51b8
ORGANIZER:mailto:[email protected]
END:VFREEBUSY
END:VCALENDAR
</cal:calendar-data>
  </cal:response>
</cal:schedule-response>

st3iny avatar Jun 20 '25 08:06 st3iny