[QA] Encryption cannot be enabled on Ubuntu 22.04

[jnweiger]

Seen in 10.10.0 RC2 with encryption 1.5.1 on ubuntu-22.04

Install using ondej ppa:

    apt install -y software-properties-common
    LC_ALL=C.UTF-8 add-apt-repository --yes ppa:ondrej/php
    LC_ALL=C.UTF-8 add-apt-repository --yes ppa:ondrej/apache2
    apt update
    apt install -y libapache2-mod-php7.4 php7.4-imagick php7.4-common php7.4-curl php7.4-gd php7.4-imap php7.4-intl
    apt install -y php7.4-ldap php7.4-pgsql php7.4-json php7.4-mbstring php7.4-mysql php7.4-sqlite3 php7.4-ssh2
    apt install -y php7.4-xml php7.4-zip php7.4-apcu php7.4-redis php7.4-gmp

• visit the admin settings, encryption: [x] enable encryption
• go to admin settings, apps, disabled: enable encryption
• go back to the admin settings, encryption - > select master key encryption, permanently.
• logout, re-login the admin user.
• use texteditor to edit a text file:
• try upload a jpeg file

Excerpt from the server log:

{"reqId":"Yn1F84Gdy-kzeBUhQtOonwAAAA0","level":2,"time":"2022-05-12T17:37:55+00:00","remoteAddr":"2.247.254.58","user":"admin","app":"core","method":"PUT","url":"\/remote.php\/dav\/files\/admin\/Docume
nts\/pinions-10-14.png","message":"ignoring lock release with type 1 for files\/d00679c268ac7e196606a02c2e166b5b. Lock hasn't been acquired before"}
{"reqId":"Yn1F84Gdy-kzeBUhQtOonwAAAA0","level":4,"time":"2022-05-12T17:37:55+00:00","remoteAddr":"2.247.254.58","user":"admin","app":"webdav","method":"PUT","url":"\/remote.php\/dav\/files\/admin\/Docu
ments\/pinions-10-14.png","message":"Exception: HTTP\/1.1 503 Encryption not ready: multikeyencryption failed error:0480006C:PEM routines::no start line: {\"Exception\":\"Sabre\\\\DAV\\\\Exception\\\\S
erviceUnavailable\",\"Message\":\"Encryption not ready: multikeyencryption failed error:0480006C:PEM routines::no start line\",\"Code\":0,\"Trace\":\"#0 \\\/var\\\/www\\\/owncloud\\\/apps\\\/dav\\\/lib
\\\/Connector\\\/Sabre\\\/File.php(243): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->convertToSabreException()\\n#1 \\\/var\\\/www\\\/owncloud\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/Directory.php
(173): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->put()\\n#2 \\\/var\\\/www\\\/owncloud\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(1098): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\Director
y->createFile()\\n#3 \\\/var\\\/www\\\/owncloud\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/CorePlugin.php(504): Sabre\\\\DAV\\\\Server->createFile()\\n#4 \\\/var\\\/www\\\/owncloud\\\/lib\\\/c
omposer\\\/sabre\\\/event\\\/lib\\\/WildcardEmitterTrait.php(89): Sabre\\\\DAV\\\\CorePlugin->httpPut()\\n#5 \\\/var\\\/www\\\/owncloud\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(47
2): Sabre\\\\DAV\\\\Server->emit()\\n#6 \\\/var\\\/www\\\/owncloud\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(253): Sabre\\\\DAV\\\\Server->invokeMethod()\\n#7 \\\/var\\\/www\\\/own
cloud\\\/apps\\\/dav\\\/lib\\\/Server.php(349): Sabre\\\\DAV\\\\Server->start()\\n#8 \\\/var\\\/www\\\/owncloud\\\/apps\\\/dav\\\/appinfo\\\/v2\\\/remote.php(31): OCA\\\\DAV\\\\Server->exec()\\n#9 \\\/
var\\\/www\\\/owncloud\\\/remote.php(165): require_once('\\\/var\\\/www\\\/ownclo...')\\n#10 {main}\",\"File\":\"\\\/var\\\/www\\\/owncloud\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/File.php\",\"L
ine\":696}"}
{"reqId":"Yn1F84Gdy-kzeBUhQtOonwAAAA0","level":4,"time":"2022-05-12T17:37:55+00:00","remoteAddr":"2.247.254.58","user":"admin","app":"webdav","method":"PUT","url":"\/remote.php\/dav\/files\/admin\/Docu
ments\/pinions-10-14.png","message":"Caused by: {\"Exception\":\"OCA\\\\Encryption\\\\Exceptions\\\\MultiKeyEncryptException\",\"Message\":\"multikeyencryption failed error:0480006C:PEM routines::no st
art line\",\"Code\":0,\"Trace\":\"#0 \\\/var\\\/www\\\/owncloud\\\/apps\\\/encryption\\\/lib\\\/Crypto\\\/Encryption.php(289): OCA\\\\Encryption\\\\Crypto\\\\Crypt->multiKeyEncrypt()\\n#1 \\\/var\\\/ww
w\\\/owncloud\\\/lib\\\/private\\\/Files\\\/Stream\\\/Encryption.php(424): OCA\\\\Encryption\\\\Crypto\\\\Encryption->end()\\n#2 [internal function]: OC\\\\Files\\\\Stream\\\\Encryption->stream_close()
\\n#3 \\\/var\\\/www\\\/owncloud\\\/apps\\\/files_external\\\/3rdparty\\\/icewind\\\/streams\\\/src\\\/Wrapper.php(96): fclose()\\n#4 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Files\\\/Stream\\\/
Checksum.php(174): Icewind\\\\Streams\\\\Wrapper->stream_close()\\n#5 [internal function]: OC\\\\Files\\\\Stream\\\\Checksum->stream_close()\\n#6 \\\/var\\\/www\\\/owncloud\\\/apps\\\/dav\\\/lib\\\/Con
nector\\\/Sabre\\\/File.php(209): fclose()\\n#7 \\\/var\\\/www\\\/owncloud\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/Directory.php(173): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->put()\\n#8 \\\/var\\\/www\\\/owncloud\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(1098): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\Directory->createFile()\\n#9 \\\/var\\\/www\\\/owncloud\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/CorePlugin.php(504): Sabre\\\\DAV\\\\Server->createFile()\\n#10 \\\/var\\\/www\\\/owncloud\\\/lib\\\/composer\\\/sabre\\\/event\\\/lib\\\/WildcardEmitterTrait.php(89): Sabre\\\\DAV\\\\CorePlugin->httpPut()\\n#11 \\\/var\\\/www\\\/owncloud\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(472): Sabre\\\\DAV\\\\Server->emit()\\n#12 \\\/var\\\/www\\\/owncloud\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(253): Sabre\\\\DAV\\\\Server->invokeMethod()\\n#13 \\\/var\\\/www\\\/owncloud\\\/apps\\\/dav\\\/lib\\\/Server.php(349): Sabre\\\\DAV\\\\Server->start()\\n#14 \\\/var\\\/www\\\/owncloud\\\/apps\\\/dav\\\/appinfo\\\/v2\\\/remote.php(31): OCA\\\\DAV\\\\Server->exec()\\n#15 \\\/var\\\/www\\\/owncloud\\\/remote.php(165): require_once('\\\/var\\\/www\\\/ownclo...')\\n#16 {main}\",\"File\":\"\\\/var\\\/www\\\/owncloud\\\/apps\\\/encryption\\\/lib\\\/Crypto\\\/Crypt.php\",\"Line\":710}"}
{"reqId":"Yn1F84Gdy-kzeBUhQtOonwAAAA0","level":1,"time":"2022-05-12T17:37:55+00:00","remoteAddr":"2.247.254.58","user":"admin","app":"core","method":"PUT","url":"\/remote.php\/dav\/files\/admin\/Documents\/pinions-10-14.png","message":"cleaning stray exclusive locks for files\/d00679c268ac7e196606a02c2e166b5b"}

[jnweiger]

Probably a setup issue. It works when installing on Ubuntu 20.04

Encryption still works fine on Ubuntu 20.04, after installing all the above mentioned packages from ondrej ppa.

[pako81]

I can indeed reproduce the issue. Steps taken:

sudo -u www-data php occ app:enable encryption
sudo -u www-data php occ encryption:enable
sudo -u www-data php occ encryption:select-encryption-type masterkey
sudo -u www-data php occ encryption:encrypt-all --yes

Last step triggers an error:

root@Pasquale-Ubuntu-22:/var/www/html/owncloud1091# sudo -u www-data php occ encryption:encrypt-all --yes

You are about to encrypt all files stored in your ownCloud installation.
Depending on the number of available files, and their size, this may take quite some time.
Please ensure that no user accesses their files during this time!
Note: The encryption module you use determines which files get encrypted.


Encrypt all files with the Default encryption module
====================================================


Use master key to encrypt all files.


Start to encrypt users files
----------------------------



 %message% 
 [>---------------------------]
In Crypt.php line 710:
                                                                        
 multikeyencryption failed error:0480006C:PEM routines::no start line

[pako81]

Error from owncloud.log:

{"reqId":"gI0KwBkGEP8zOkRpW35s","level":3,"time":"2022-05-20T13:31:41+00:00","remoteAddr":"","user":"--","app":"PHP","method":"--","url":"--","message":"fclose(): supplied resource is not a valid stream resource at \/var\/www\/html\/owncloud1091\/lib\/private\/Files\/Storage\/Wrapper\/Encryption.php#815"}

[pako81]

The problem is due to the fact that Ubuntu 22.04 comes with openssl v3.0.2 per default and in this version legacy ciphers (which we are still using for server-side encryption) are now disabled.

This can be workarounded by editing the openssl.config file this way:

# List of providers to load
[provider_sect]
default = default_sect
legacy = legacy_sect
# The fips section name should match the section name inside the
# included fipsmodule.cnf.
# fips = fips_sect

# If no providers are activated explicitly, the default one is activated implicitly.
# See man 7 OSSL_PROVIDER-default for more details.
#
# If you add a section explicitly activating any other provider(s), you most
# probably need to explicitly activate the default provider, otherwise it
# becomes unavailable in openssl.  As a consequence applications depending on
# OpenSSL may not work correctly which could lead to significant system
# problems including inability to remotely access the system.
[default_sect]
activate = 1

[legacy_sect]
activate = 1

this will make encryption to go through and correctly encrypt all files. It may require documentation if we decide to go for this approach (I don't see any other alternative rather than manually compiling an older openssl version on Ubuntu 22.04, which is a no-go IMHO) @mmattel FYI

[mmattel]

We need to document what legacy cyphers are and what it means to customers in terms of security.

[mmattel]

Here is some background info:

• U20.04 comes with openSSL 1.1.1f
• U22.04 comes with openSSL 3.0.2
• As we can see, there is a major version change of openSSL 1.x --> 3.x Some ciphers we use have been set to legacy and should not be used anymore
• See the openSSL migration docs, especially the Legacy-Algorithms section

@jnweiger we need to do proper testing and update documentation in several locations of our admin docs accordingly!

@pmaier1 this is a serious topic and does not only affect U22.04 but also other distros when they go for openSSL 3.x. My recommandation is, that we do a two step task:

1. quick fix: QA this issue and do proper documentation - just to keep existing OC10 systems up and running when an OS has openSSl 3.x
2. longterm fix: fix the underlaying issue of legacy ciphers in core/encryption by creating a manually triggered occ migration command set (like occ cipher list/check/update...) to manage ciphers in OC10, remove the legacy cipher enablement in the openSSL config and update the documentation again. This also means a new ownCloud or encryption version...

[jnweiger]

Especially with other LTS Linux Platforms, we need to find, which other platforms are affected by openssl cypher deprecations.

• Ubuntu 21.10 has: 1.1.1l
• Ubuntu 22.04 has: 3.0.2
• Debian 11 has: 1.1.1n-0+deb11u2
• SLES 15 SP3 has: 1.1.1 or 1.0.2 (Module Legacy)
• CentOS 8 Stream has: 1.1.1k or EPEL 3.0.1
• CentOS 9 Stream has: 3.0.1
• Fedora 36 has: 3.0.2 and 1.1.1n

Migration from one cypher to another will be very compute intensive, and must be well planned.

[IljaN]

See the openSSL migration docs, especially the Legacy-Algorithms section

This is probably not due to deprecated algorithms as we never use MD2 or DES (maybe our libraries do?).

We do use various openssl apis throughout our codebase. grep -R \openssl_ * should be a good starting point.

[pako81]

@IljaN correct, https://github.com/owncloud/encryption/blob/v1.5.1/lib/Crypto/Crypt.php#L696 and https://github.com/owncloud/encryption/blob/v1.5.1/lib/Crypto/Crypt.php#L671 are the problematic calls AFAICT

[IljaN]

@pako81 Do both calls return this error? Do you know? multikeyencryption failed error:0480006C:PEM routines::no start line

[pako81]

@IljaN https://github.com/owncloud/encryption/blob/v1.5.1/lib/Crypto/Crypt.php#L696 is first called when encrypting files with master key and, yes, it returns an multikeyencryption failed error:0480006C:PEM routines::no start line error

[pako81]

while https://github.com/owncloud/encryption/blob/v1.5.1/lib/Crypto/Crypt.php#L671 is called when converting to plain text to read the file. I did not test it but I assume it will trigger the same error.

[IljaN]

Hmhh... I wonder if we need to change our encryption key format and provide a migration as @mmattel already hinted because PEM routines::no start line sounds like openssl v3 can't read our encryption keys due to some file-format issue anymore. Needs more digging...

[IljaN]

Are some of our key-files missing "start lines" ? https://snippets.aktagon.com/snippets/543-how-to-fix-pem-read-bio-no-start-line-error-nginx-error

Adding those might fix this, however we then need to check every site where we read keys, as we might not use openssl_* in some places but read the files with fopen and might not expect a start line there.

[IljaN]

@pako81 Was this an upgraded instance or a fresh-one? Because I would assume that if the keys were initially written with openssl v3 it would automatically add those start-lines (or maybe not :thinking: ;))

[pako81]

@IljaN fresh-one with no files except the standard ones.

[IljaN]

There is more :cry: (Create new file, try to save)

multikeyencryption failed error:0308010C:digital envelope routines::unsupported\",\"Code\":0,\"Trace\":\"#0 \\\/home\\\/ilja\\\/code\\\/core\\\/apps\\\/encryption\\\/lib\\\/Crypto\\\/Encryption.php(293)

[phil-davis]

There is now an owncloudci/php:7.4-ubuntu22.04 docker image. PR #40123 runs the automated core test suites with that. cliEncryption suite fails - there are some core tests that have been set up to run enable encryption and run some basic encryption tests in the core CI.

See comment https://github.com/owncloud/core/pull/40123#issuecomment-1148545513

So this confirms that the issue happens in an automated test environment.

If we sort out a solution that combines encryption settings, code changes, whatever, then we can apply the suggested solution to the automated tests and quite easily know if it works.

[mmattel]

There is now an owncloudci/php:7.4-ubuntu22.04 docker image

@phil-davis you may want to see the comment regarding a needed openSSL config change https://github.com/owncloud/encryption/issues/342#issuecomment-1146054526

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 10 days if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically closed.