server [Bug]: Error PHP unserialize

⚠️ This issue respects the following points: ⚠️

[X] This is a bug, not a question or a configuration/webserver/proxy issue.
[X] This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
[X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
[X] I agree to follow Nextcloud's Code of Conduct.

Bug description

Hi I would like to report a repeated error I can find in NC logs. I'm afraid I can't evaluate its consequences, many errors arrived at the same time: locked files, antivirus, SQL and redis server.

Steps to reproduce

find logs in NC Logging:

[PHP] Error: unserialize(): Error at offset 40 of 43 bytes at /var/www/html/apps/dav/lib/DAV/CustomPropertiesBackend.php#574
	PROPFIND /remote.php/dav/addressbooks/users/UserName/
	from 185.252.235.96 by UserName at 8 sept. 2024, 19:22:05

Expected behavior

no error

Nextcloud Server version

29

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.2

Web server

Nginx

Database engine version

PostgreSQL

Is this bug present after an update or on a fresh install?

Updated from a MINOR version (ex. 28.0.1 to 28.0.2)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

[X] Default user-backend (database)
[ ] LDAP/ Active Directory
[ ] SSO - SAML
[ ] Other

Configuration report

No response

List of activated Apps

No response

Nextcloud Signing status

No response

Nextcloud Logs

{"reqId":"YizWhMVEux6WGmu7Clxp","level":3,"time":"2024-09-08T17:22:05+00:00","remoteAddr":"185.252.235.96","user":"UserName","app":"PHP","method":"PROPFIND","url":"/remote.php/dav/addressbooks/users/UserName/","message":"unserialize(): Error at offset 40 of 43 bytes at /var/www/html/apps/dav/lib/DAV/CustomPropertiesBackend.php#574","userAgent":"Mac OS X/10.15.7 (19H2026) AddressBookCore/1","version":"29.0.4.1","data":{"app":"PHP"},"id":"66e090d696463"}



### Additional info


#Server configuration detail
Operating system: Linux 5.10.0-32-amd64 [#](https://notre.rez0.net/s/ztLijxYSegjCmjt#h-server-configuration-detail)1 SMP Debian 5.10.223-1 (2024-08-10) x86_64

Webserver: Apache/2.4.62 (Unix) (fpm-fcgi)

Database: pgsql PostgreSQL 16.3 on x86_64-pc-linux-musl, compiled by gcc (Alpine 13.2.1_git20240309) 13.2.1 20240309, 64-bit

PHP version: 8.2.21

Modules loaded: Core, date, libxml, openssl, pcre, sqlite3, zlib, ctype, curl, dom, fileinfo, filter, hash, iconv, json, mbstring, SPL, session, PDO, pdo_sqlite, bz2, posix, random, readline, Reflection, standard, SimpleXML, tokenizer, xml, xmlreader, xmlwriter, mysqlnd, cgi-fcgi, apcu, bcmath, Phar, exif, ftp, gd, gmp, igbinary, imagick, imap, intl, ldap, memcached, pcntl, pdo_pgsql, pgsql, redis, smbclient, sodium, sysvsem, zip, libsmbclient, Zend OPcache

Nextcloud version: 29.0.4 - 29.0.4.1

Updated from an older Nextcloud/ownCloud or fresh install:

Where did you install Nextcloud from: unknown

<details><summary>Signing status</summary>

[]
</details>

<details><summary>List of activated apps</summary>


Enabled:
 - activity: 2.21.1
 - admin_audit: 1.19.0
 - announcementcenter: 6.8.1
 - auto_groups: 1.5.3
 - bruteforcesettings: 2.9.0
 - calendar: 4.7.16
 - cfg_share_links: 5.1.3
 - circles: 29.0.0-dev
 - cloud_federation_api: 1.12.0
 - collectives: 2.14.3
 - comments: 1.19.0
 - contacts: 6.0.0
 - contactsinteraction: 1.10.0
 - dashboard: 7.9.0
 - dav: 1.30.1
 - deck: 1.13.1
 - event_update_notification: 2.4.0
 - external: 5.4.0
 - federatedfilesharing: 1.19.0
 - files: 2.1.0
 - files_antivirus: 5.5.7
 - files_downloadlimit: 2.0.0
 - files_pdfviewer: 2.10.0
 - files_reminders: 1.2.0
 - files_sharing: 1.21.0
 - files_trashbin: 1.19.0
 - files_versions: 1.22.0
 - forms: 4.2.4
 - group_default_quota: 0.1.10
 - groupfolders: 17.0.3
 - integration_excalidraw: 2.2.0
 - integration_youtube: 0.3.0
 - logreader: 2.14.0
 - lookup_server_connector: 1.17.0
 - mail: 3.7.8
 - money: 0.28.0
 - nextcloud-aio: 0.6.0
 - nextcloud_announcements: 1.18.0
 - notes: 4.10.1
 - notifications: 2.17.0
 - notify_push: 0.7.0
 - oauth2: 1.17.0
 - password_policy: 1.19.0
 - polls: 7.2.2
 - provisioning_api: 1.19.0
 - quota_warning: 1.20.0
 - registration: 2.4.0
 - related_resources: 1.4.0
 - richdocuments: 8.4.6
 - settings: 1.12.0
 - sharebymail: 1.19.0
 - side_menu: 3.13.1
 - spreed: 19.0.8
 - support: 1.12.0
 - suspicious_login: 7.0.0
 - tasks: 0.16.1
 - terms_of_service: 2.5.0
 - text: 3.10.1
 - theming: 2.4.0
 - timemanager: 0.3.15
 - twofactor_backupcodes: 1.18.0
 - unroundedcorners: 1.1.3
 - user_status: 1.9.0
 - viewer: 2.3.0
 - welcome: 1.2.0
 - workflowengine: 2.11.0
Disabled:
 - encryption
 - federation: 1.17.0
 - files_external
 - firstrunwizard: 2.14.0
 - impersonate: 1.16.0
 - photos: 2.3.0
 - privacy: 1.13.0
 - recommendations: 1.4.0
 - serverinfo: 1.16.0
 - survey_client: 1.13.0
 - systemtags: 1.18.0
 - twofactor_totp: 7.0.0
 - user_ldap
 - weather_status: 1.5.0

</details>

<details><summary>Configuration (config/config.php)</summary>


{
    "memcache.local": "\\OC\\Memcache\\APCu",
    "apps_paths": [
        {
            "path": "\/var\/www\/html\/apps",
            "url": "\/apps",
            "writable": false
        },
        {
            "path": "\/var\/www\/html\/custom_apps",
            "url": "\/custom_apps",
            "writable": true
        }
    ],
    "memcache.distributed": "\\OC\\Memcache\\Redis",
    "memcache.locking": "\\OC\\Memcache\\Redis",
    "redis": {
        "host": "***REMOVED SENSITIVE VALUE***",
        "password": "***REMOVED SENSITIVE VALUE***",
        "port": 6379
    },
    "overwriteprotocol": "https",
    "passwordsalt": "***REMOVED SENSITIVE VALUE***",
    "secret": "***REMOVED SENSITIVE VALUE***",
    "trusted_domains": [
        "localhost",
        "notre.rez0.net",
        "kolab.koraland.net"
    ],
    "datadirectory": "***REMOVED SENSITIVE VALUE***",
    "skeletondirectory": "\/var\/lib\/docker\/volumes\/nextcloud_aio_nextcloud\/_data\/skeleton",
    "dbtype": "pgsql",
    "version": "29.0.4.1",
    "dbname": "***REMOVED SENSITIVE VALUE***",
    "dbhost": "***REMOVED SENSITIVE VALUE***",
    "dbport": "",
    "dbtableprefix": "oc_",
    "dbuser": "***REMOVED SENSITIVE VALUE***",
    "dbpassword": "***REMOVED SENSITIVE VALUE***",
    "installed": true,
    "instanceid": "***REMOVED SENSITIVE VALUE***",
    "check_data_directory_permissions": true,
    "maintenance": false,
    "loglevel": 2,
    "log_type": "file",
    "logfile": "\/var\/www\/html\/data\/nextcloud.log",
    "log_rotate_size": "10485760",
    "log.condition": {
        "apps": [
            "admin_audit"
        ]
    },
    "preview_max_x": "2048",
    "preview_max_y": "2048",
    "jpeg_quality": "60",
    "enabledPreviewProviders": {
        "1": "OC\\Preview\\Image",
        "2": "OC\\Preview\\MarkDown",
        "3": "OC\\Preview\\MP3",
        "4": "OC\\Preview\\TXT",
        "5": "OC\\Preview\\OpenDocument",
        "6": "OC\\Preview\\Movie",
        "0": "OC\\Preview\\Imaginary"
    },
    "enable_previews": true,
    "upgrade.disable-web": true,
    "trashbin_retention_obligation": "auto, 30",
    "versions_retention_obligation": "auto, 30",
    "activity_expire_days": "30",
    "simpleSignUpLink.shown": false,
    "share_folder": "\/Shared",
    "one-click-instance": true,
    "one-click-instance.user-limit": 100,
    "one-click-instance.link": "https:\/\/nextcloud.com\/all-in-one\/",
    "htaccess.RewriteBase": "\/",
    "files_external_allow_create_new_local": true,
    "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
    "preview_imaginary_url": "***REMOVED SENSITIVE VALUE***",
    "default_language": "fr",
    "default_locale": "fr_FR",
    "default_phone_region": "FR",
    "mail_sendmailmode": "smtp",
    "mail_from_address": "***REMOVED SENSITIVE VALUE***",
    "mail_domain": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpauthtype": "LOGIN",
    "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpport": "465",
    "mail_smtpauth": 1,
    "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
    "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
    "allow_local_remote_servers": true,
    "updatedirectory": "\/nc-updater",
    "overwritehost": "notre.rez0.net",
    "overwrite.cli.url": "https:\/\/notre.rez0.net\/",
    "updater.release.channel": "stable",
    "mail_smtpmode": "smtp",
    "mail_smtpsecure": "ssl",
    "upgrade.cli-upgrade-link": "https:\/\/github.com\/nextcloud\/all-in-one\/discussions\/2726",
    "davstorage.request_timeout": 3600,
    "dbpersistent": false,
    "appsallowlist": false,
    "maintenance_window_start": 100,
    "preview_imaginary_key": "***REMOVED SENSITIVE VALUE***",
    "defaultapp": "",
    "auth.bruteforce.protection.enabled": true,
    "ratelimit.protection.enabled": true
}
</details>

Cron Configuration: Array
(
[backgroundjobs_mode] => cron
[lastcron] => 1725993214
)

External storages: files_external is disabled

Encryption: no

User-backends:

OC\User\Database

Talk configuration:

STUN servers

185.252.235.96:443

TURN servers

turn:185.252.235.96:3478 - udp,tcp

Signaling servers (mode: default):

SIP dialin is disabled

SIP dialout is disabled

https://notre.rez0.net/standalone-signaling/ - 1.3.2~docker

Recording servers:

Recording is enabled

Recording consent is set to "default"

no recording server configured

Browser: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36

Sep 10 '24 18:09 punkyard

I stumbled upon a similiar issue and just post my findings here instead of making a new bug report.

The issue is that postgres does not allow NUL bytes in TEXT fields, however, CustomPropertiesBackend uses serialize to store non scalar values. Sabre's ResourceType has a private field in it, and serialize documentation states:

Object's private members have the class name prepended to the member name; protected members have a '*' prepended to the member name. These prepended values have null bytes on either side.

Also,

Note that this is a binary string which may include null bytes, and needs to be stored and handled as such. For example, serialize() output should generally be stored in a BLOB field in a database, rather than a CHAR or TEXT field.

The problem now is that we try to store that serialized value as-is, which Postgres doesn't support. One way to fix it would be to store non-text values base64 encoded, or alternatively, change the type of propertyvalue from TEXT to BLOB (or BYTEA) and store normal text as UTF-8 encoded blob and objects as-is.

Obviously this would require a database migration.

Sep 18 '24 16:09 brakhane

Also see https://github.com/nextcloud/server/issues/37754#issuecomment-1613361252

Sep 19 '24 21:09 joshtrichards

hi @joshtrichards thanks for your reply

I have looked at #37754 and it seems to be about column type when this issue here could be considered as a quantity error what do you think?

Sep 28 '24 14:09 punkyard

Also experiencing this issue. I'm running Nextcloud Hub 9 (30.0.0) though the docker image Nextcloud AIO v9.6.0.

I see the below message spammed a ton of times in my log, right after I added my subscribed to my Nextcloud calendar via URL to a Google calendar.

[PHP] Error: unserialize(): Error at offset 61 of 64 bytes at /var/www/html/apps/dav/lib/DAV/CustomPropertiesBackend.php#560
	PROPFIND /remote.php/dav/calendars/[USER_REDACTED]/
	from [IP_REDACTED] by [USER_REDACTED] at Oct 10, 2024, 7:13:59 PM

Oct 10 '24 23:10 starlingfire

Also experiencing this issue. I'm running Nextcloud Hub 9 (30.0.0) though the docker image Nextcloud AIO v9.6.0.

I see the below message spammed a ton of times in my log, right after I added my subscribed to my Nextcloud calendar via URL to a Google calendar.
[PHP] Error: unserialize(): Error at offset 61 of 64 bytes at /var/www/html/apps/dav/lib/DAV/CustomPropertiesBackend.php#560
	PROPFIND /remote.php/dav/calendars/[USER_REDACTED]/
	from [IP_REDACTED] by [USER_REDACTED] at Oct 10, 2024, 7:13:59 PM

Same here, also on Hub 9 with Nextcloud AIO. I also have a Google Calendar subscription so that must be it. Calendar entries for that are showing up correctly tho

Oct 21 '24 18:10 phil-lipp

cc @miaulalala @SebastianKrupinski

Oct 23 '24 17:10 susnux

Thanks for the heads up @susnux will have a look.

Oct 23 '24 20:10 SebastianKrupinski

Can confirm this issue: [PHP] Error: unserialize(): Error at offset 61 of 64 bytes at /var/www/html/apps/dav/lib/DAV/CustomPropertiesBackend.php#560 PROPFIND /remote.php/dav/calendars/[USER]/ from [IP] by [USER] at 20 Nov 2024, 22:48:52

Running Hub9 30.0.2 version with Calendar 5.0.5 on TrueNas Scale 24.10 in docker with postgress

Nov 20 '24 22:11 kabatp

Hi @kabatp is there any more details on this error in the Nextcloud log?

Nov 21 '24 04:11 SebastianKrupinski

@SebastianKrupinski all I can find is

{ "reqId": "jt9TMExds0xZthdCzsbl", "level": 3, "time": "2024-11-21T06:39:52+00:00", "remoteAddr": "REDACTED", "user": "REDACTED", "app": "PHP", "method": "PROPFIND", "url": "/remote.php/dav/calendars/REDACTED/", "message": "unserialize(): Error at offset 61 of 64 bytes at /var/www/html/apps/dav/lib/DAV/CustomPropertiesBackend.php#560", "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:132.0) Gecko/20100101 Firefox/132.0", "version": "30.0.2.2", "data": { "app": "PHP" }, "id": "673ed5b8f05f3" }

As far as I can tell the error started popping up when I tried to set up calendars and contacts. Contacts were manually migrated which means this should not be caused by VCF compatibility issues. Also, I tried to delete all calendars, but the error was still there which I am assuming the problem is with the contacts. The error shows circa every 30 seconds.

The error message points to:

private function decodeValueFromDatabase(string $value, int $valueType) {
                switch ($valueType) {
                        case self::PROPERTY_TYPE_XML:
                                return new Complex($value);
                        case self::PROPERTY_TYPE_HREF:
                                return new Href($value);
                        case self::PROPERTY_TYPE_OBJECT:
                                return unserialize($value);
                        case self::PROPERTY_TYPE_STRING:
                        default:
                                return $value;
                }
        }

specifically to this part

case self::PROPERTY_TYPE_OBJECT:
                                return unserialize($value);

I have tried to play around in the database but couldn't find any data that could cause the issue. Also, this should not be a setup issue as it occurs only with added contacts - when I delete the contacts the issue is no longer there. All of the contacts have been manually created one by one which indicates bad handling of empty fields - as postgress is not good with handling NUL chars as mentioned in https://github.com/nextcloud/server/issues/37754#issuecomment-1589463097

Nov 21 '24 06:11 kabatp

@SebastianKrupinski I exported contacts from Nextcloud, deleted them and imported them back and it looks like the issue went away. I believe the export contains only fields that are filled which means the issue should be connected to the default fields that are shown in the GUI when creating contacts

Nov 21 '24 08:11 kabatp

@SebastianKrupinski I exported contacts from Nextcloud, deleted them and imported them back and it looks like the issue went away. I believe the export contains only fields that are filled which means the issue should be connected to the default fields that are shown in the GUI when creating contacts

I am taking back this statement as the issue reappeared after one hour after importing the contacts back

Nov 21 '24 09:11 kabatp

I have tried to play around in the database but couldn't find any data that could cause the issue

There's a workaround that involves deleting the offending record, which is what I did.

Connect to the container running the database. I'm running the AIO, so it's in its own container:

sudo docker exec -it nextcloud-aio-database bash

Connect to the database:

psql -U "oc_$POSTGRES_USER" -d $POSTGRES_DB -h localhost

View the offending records:

SELECT * FROM oc_properties WHERE propertyvalue like concat('%', 0x00, '%');

Update the offending records:

UPDATE oc_properties SET propertyvalue = replace(propertyvalue, 0x00, '') WHERE propertyvalue like concat('%', 0x00, '%');

In my case, running the UPDATE command in Step 5 did not work (nor do I recall if the SELECT statement in Step 4), so I ended up just deleting all records where the valuetype = 3:

SELECT * FROM oc_properties WHERE valuetype = 3;
DELETE FROM oc_properties WHERE valuetype = 3;

My assumption would be that you'd need to run this any time you added a new calendar or contact list. Basically anything that creates a record in oc_properties with valuetype = 3.

Nov 27 '24 14:11 starlingfire

@starlingfire thanks for this, on first look in seems to fix the issue but I will monitor it and update all of you. The row with valuetype = 3 and strange propertyvalue was holiday calendar

id | userid |                     propertypath                     |               propertyname               | propertyvalue | valuetype 
18 | REDACTED| calendars/REDACTED/holidays-in-slovakia                | {urn:ietf:params:xml:ns:caldav}schedule-calendar-transp | O:48:"Sabre\CalDAV\Xml\Property\ScheduleCalendarTransp":1:{s:8:" |         3

It was added directly via Nextcloud calendar - in my understanding it is taken from Mozilla Thunderbird calendars

Nov 27 '24 18:11 kabatp

I downloaded the holiday calendar from official Thunderbird site - in that case, the valuetype is set to 1 and there is no more issue. This suggests to me that this is purely a Nextcloud issue in how it handles the holiday calendars.

Nov 27 '24 18:11 kabatp

@kabatp which holiday calendar was causing this?

@starlingfire was your error also caused by a holiday calendar? Or the google calendar?

Nov 27 '24 19:11 SebastianKrupinski

@SebastianKrupinski for me it was Slovak holiday calendar. For resolving I tried random calendars - holidays in romania and also czech, but these two are stored with propertyvalue 1 and not cause any issue.

Nov 27 '24 19:11 kabatp

Found it! Damn little bug was hard to find. Been trying to find it for months.

Tested - Confirmed

Add subscription Calendar, through the Calendar app. I used "https://www.thunderbird.net/media/caldata/autogen/CanadaHolidays.ics"
Edit subscribed calendar name, and press save
Refresh calendar app
Check logs.

{
  "reqId": "BUiQH11laLmZ1WSNxmTX",
  "level": 3,
  "time": "2024-11-27T21:02:01+00:00",
  "remoteAddr": "127.0.0.1",
  "user": "user1",
  "app": "PHP",
  "method": "PROPFIND",
  "url": "/remote.php/dav/calendars/user1/",
  "message": "unserialize(): Error at offset 61 of 64 bytes at /var/www/nextcloud/master/apps/dav/lib/DAV/CustomPropertiesBackend.php#537",
  "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:133.0) Gecko/20100101 Firefox/133.0",
  "version": "31.0.0.5",
  "data": {
    "app": "PHP"
  }
}

Issue

Property "{urn:ietf:params:xml:ns:caldav}schedule-calendar-transp" is being marked as a object, this then gets encoded in ::encodeValueForDatabase() as an object. Issue is in OCA\DAV\DAV\CustomPropertiesBackend::encodeValueForDatabase() line 529. The serialization of the object causes null characters in the string produced.

Possible Resolutions

Serialize only the value of the object $value->getValue()
Replace serialization with json_encode() / json_decode()
Replace null characters after serialization str_replace("\0", '', $serialized);

Nov 27 '24 22:11 SebastianKrupinski

thank you @SebastianKrupinski and congratulations

Will it be soon integrated in a future version then?

Nov 27 '24 23:11 punkyard

Issue is in OCA\DAV\DAV\CustomPropertiesBackend::encodeValueForDatabase() line 529.

Could you link to the problematic code using a master commit? Here is what I get right now: https://github.com/nextcloud/server/blob/659cd12a8a31eba3d1896bab8e14aab2243eb327/apps/dav/lib/DAV/CustomPropertiesBackend.php#L529

What is the value of the property in your case? XML?

Example:

 <C:schedule-calendar-transp
      xmlns:C="urn:ietf:params:xml:ns:caldav">
   <C:opaque/>
 </C:schedule-calendar-transp>

https://www.rfc-editor.org/rfc/rfc6638.txt

Nov 28 '24 07:11 ChristophWurst

@ChristophWurst

Here is what is causing the error...

https://github.com/nextcloud/server/blob/379f575c25cdf4769d5c019394e73ac8b8f46385/apps/dav/lib/DAV/CustomPropertiesBackend.php#L522

Yes this is the property that has the issue

Example:

 <C:schedule-calendar-transp
      xmlns:C="urn:ietf:params:xml:ns:caldav">
   <C:opaque/>
 </C:schedule-calendar-transp>

rfc-editor.org/rfc/rfc6638.txt

Nov 28 '24 12:11 SebastianKrupinski

Shouldn't that go through https://github.com/nextcloud/server/blob/659cd12a8a31eba3d1896bab8e14aab2243eb327/apps/dav/lib/DAV/CustomPropertiesBackend.php#L514-L516 if the value is XML?

Nov 28 '24 12:11 ChristophWurst

Shouldn't that go through

server/apps/dav/lib/DAV/CustomPropertiesBackend.php

One would think that, but no it is marked as PROPERTY_TYPE_OBJECT (value 3)

Nov 28 '24 12:11 SebastianKrupinski

That happens because of https://github.com/nextcloud/server/blob/659cd12a8a31eba3d1896bab8e14aab2243eb327/apps/dav/lib/DAV/CustomPropertiesBackend.php#L521, right? I think you have to trace this back a bit and find the caller of encodeValueForDatabase. Should the $value parameter be of an XML type perhaps?

Nov 28 '24 12:11 ChristophWurst

I think you have to trace this back a bit and find the caller of encodeValueForDatabase. Should the $value parameter be of an XML type perhaps?

The value gets determined in the same function: https://github.com/nextcloud/server/blob/dd101dd0f70fc740106c6db30b0742e4db772b08/apps/dav/lib/DAV/CustomPropertiesBackend.php#L512

So value 3 is technically correct. The values are determined by the instance of a class. XML properties are instances of "Sabre\DAV\Xml\Property\Complex" which is a instance of "Sabre\Xml\Element\XmlFragment" which in turn is a instance of "Sabre\Xml\Element"

But "Sabre\CalDAV\Xml\Property\ScheduleCalendarTransp" is an instance of "Sabre\Xml\Element" directly.

That said, I think we should fix the serialization of objects either way and yes "ScheduleCalendarTransp" should probably be an instance of "Complex".

Nov 28 '24 17:11 SebastianKrupinski