Not starting anymore since v23.5.5 (arm64)

[Pilzinsel64]

Issue

I'm using the app Collabora Online Built-in CODE (arm64). Since the last update (v23.5.5) the CODE server seems to have problems to start. When downgrading to the previous version (v22.5.1301) and trigger everything works fine again.

But I need to do the downgrade each time the server restarts (basically every night) now since there is no possibility to disbale auto-updates in the snap packages.

Log

There isn't any new entry in the Nextcloud log. If you want the log anyway, please tell me.

System report

Link: https://dragoncloud.ddnss.de/index.php/s/XFe5CdWLnyBC5nf Passwort: cBKcRkWcNs

Environment infos

• Nextcloud 26.0.2 via Snap package
• Ubuntu Server 22.04 LTS
• Raspberry Pi 4 8GB

[nordlolek]

I don't know if this is related but I also have troubles with loading Collabora Online - Built-in CODE Server after upgrading to v23.5.5 . No connection to CODE server and blank page when trying to open document files. No related errors in nextcloud log (even on debug level).

Only error I get in apache log is: coolwsd: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by coolwsd)

• OS: Centos 7 ( 3.10.0-1160.90.1.el7.x86_64 )
• PHP: 8.1.20
• Nextcloud version: 26.0.2
• APP: Collabora Online - Built-in CODE Server: v23.5.5
• APP: Nextcloud Office: v8.0.2

system-report-2023-06-13.md

[Bobmorton-TX]

same problem for me. nothing in nextcloud log.

• ROCK5B (arm64) 8GB
• OS: Armbian 23.02.2 Jammy
• Nextcloud 25.0.6 via Snap package
• Collabora v23.5.5

Collabora 22.5.1301 works without issues

[nordlolek]

Does anyone even read this issues? Some acknowledgement would be great. More and more people are reporting similar problems... I am aware this is free and opensource project but I think they at least should not be forcing this update if it is breaking things.

[timar]

coolwsd: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by coolwsd)

This looks like a serious issue, right? We build the packages for the AppImage on Ubuntu 18.04, it has glibc-2,27, while CentOS 7 has glibc-2.17. For the 22.5.x series we used Ubuntu 16.04 to build the packages. When I changed the baseline, I did not think of this use case, i.e. someone would run the deb packages on CentOS7 (via several abstraction layers, snap, Appimage, etc.). The solution could be to build deb packages on Ubuntu 16.04 again. Of course it won't solve the problems on RPi. I think those problems are unrelated to yours.

[timar]

Hmm, even Ubuntu 16.04 has glibc-2.23. The issue needs further investigation.

[Pilzinsel64]

@timar If you need any more information or want me to test something, just tell me. :)

[timar]

@timar If you need any more information or want me to test something, just tell me. :)

coolwsd from AppImage logs at weird places, can you please try to find it? In /tmp for example (find /tmp -name coolwsd.log). Also webserver's error log can tell if there is an issue with proxy.php. I tried arm64 build on AWS, it worked for me, but I know that there are many options, many different setups.

[Pilzinsel64]

@timar Sure! Here are the webserver log files: webserver-logs.zip The last line(s) of the apache_error.log and php_errors.log file may be interesting, but the time seems to not really match. 🤔

The coolwsd.log don't get generated at all when using the new version. When using v22.5.1301 there is a log file.

This is what I did from enabling the app to picking the log files:

1. I updated the app to the latest version.
2. Then I tried opening a document.
3. Lastly I opened the "Nextcloud Office" admin configuration page and clicked to the check-box "Use Built-In-CODE" to re-trigger a reconnection. After about 30 seconds it tells me that a connection to Collabora Online Server is not possible.
4. Copied over the log files.

I hope this helps! If you need anything else, just tell me.

[Pilzinsel64]

@timar Maybe the new AppImage needs a new dependency that is not installed on my system? 🤔

EDIT: I'm sorry, but I will not be able anymore to provide more details sadly. Aber long time of waiting I moved my server to a new hardware with more perfoemave and different architecture. The Built-In CODE app work here now.

[Bobmorton-TX]

unable to get Collabora to run on Rock5 since NC26 update. please see my issue here: https://github.com/nextcloud-snap/nextcloud-snap/issues/2477

[nordlolek]

I don't know if this is related but I also have troubles with loading Collabora Online - Built-in CODE Server after upgrading to v23.5.5 . No connection to CODE server and blank page when trying to open document files. No related errors in nextcloud log (even on debug level).

Only error I get in apache log is: coolwsd: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by coolwsd)

• OS: Centos 7 ( 3.10.0-1160.90.1.el7.x86_64 )
• PHP: 8.1.20
• Nextcloud version: 26.0.2
• APP: Collabora Online - Built-in CODE Server: v23.5.5
• APP: Nextcloud Office: v8.0.2

system-report-2023-06-13.md

After migration from Centos7 to Alma9 (which has support for newer versions of GLIBC) everything works ok.

[Jolopu]

I'm on a shared host and have all the latest versions of NC and collabora installed. I'm on PHP 8.2. Since 23.5.102 all versions of collabora server are not loading. When downgrading to 23.5.5 it loads. I don't know what the issue is. Maybe it has something to do with fontconfig. Anyway I will stay on 23..5.5 until a newer version is loading again.

I think this might affect also other users on shared hosts. If any developer is interested in researching this issue, I can provide access to my test server. Just give me an email.

[Pilzinsel64]

Everyone who still read this:

Did you try to install all required libraries? On Ubuntu Server 22.04 it is common that fontconfig is missing, as it looks like. At list this one library is required to get the app working. But to ensure also glibc should be installed.

See: https://github.com/CollaboraOnline/richdocumentscode#system-requirements

Just found that out while searching for a similar issue. Sadly, I can't test it myself anymore, I'm on amd64 now. :/ So please give it a try and let me know if it solved your problem! ;)

[Bobmorton-TX]

I am happy to test this on my arm (rock5) setup. can you let me know how I could install these two on armbian?

[Pilzinsel64]

Try this: sudo apt-get update && sudo apt-get install fontconfig glibc-source -y

[Bobmorton-TX]

Ich denke beide sind schon installiert.

sudo apt list fontconfig
Auflistung… Fertig
fontconfig/jammy,now 2.13.1-4.2ubuntu5 arm64  [Installiert,automatisch]
fontconfig/jammy 2.13.1-4.2ubuntu5 armhf

sudo apt list glibc-source -a

Auflistung… Fertig
glibc-source/jammy-security,jammy-security,jammy-updates,jammy-updates 2.35-0ubuntu3.4 all
glibc-source/jammy,jammy 2.35-0ubuntu3 all

[Bobmorton-TX]

habe glib installiert - hat leider nicht verbessert. akutelle version vom collabora started nicht

rock@MinkaRock5:~$ sudo apt list glibc-source -a Auflistung… Fertig glibc-source/jammy-security,jammy-security,jammy-updates,jammy-updates,now 2.35-0ubuntu3.4 all [installiert] glibc-source/jammy,jammy 2.35-0ubuntu3 all

[Jolopu]

This worked for me. Edit proxy.php on lines 152 an 153:

151 exec('( /sbin/ldconfig -p || scanelf -l ) | grep fontconfig > /dev/null 2>&1', $output, $return); 152 // if ($return) 153 // return 'no_fontconfig'; 154 return '';

Fontconfig is installed and running on my server. However Collabora still thinks it's not.

[Pilzinsel64]

@Jolopu Interesting. But we're close.

What is the output of this command on your system? ( /sbin/ldconfig -p || scanelf -l ) | grep fontconfig?

Mine is:

libfontconfig.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libfontconfig.so.1

... while libfontconfig.so.1 is the requirement.

[Pilzinsel64]

Also, what is the output of this address? https://yourcloud.com/index.php/apps/richdocuments/settings/fonts.json

[Jolopu]

{"kind":"fontconfiguration","server":"Myserver (https://mycloud.tv)","fonts":[]}

[Jolopu]

@Jolopu Interesting. But we're close.

What is the output of this command on your system? ( /sbin/ldconfig -p || scanelf -l ) | grep fontconfig?

Mine is:

libfontconfig.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libfontconfig.so.1

... while libfontconfig.so.1 is the requirement.

I'm on a shared host. Sorry.

[Pilzinsel64]

Ok, thank you anyway for tryit it out. Then, maybe another library is still missing. 🤔

[Pilzinsel64]

@timar We wrote a guide how to install Collabora Online & Built-In CODE on a fresh installed Ubuntu Server 22.04 LTS on x86_64 devices. This should be the same for ARM64.

Please, follow my steps here and you will be able to reproduce and debug the issue, I'm 100% sure!

1. Install Ubuntu Server 22.04 and allo third party dependencies.
2. Install Nextcloud via sudo snap install nextcloud
3. Follow all other steps provided here: https://github.com/nextcloud-snap/nextcloud-snap/wiki/Configure-CODE-and-Nextcloud-office-for-Nextcloud-snap

I hope you can take you the time, there are many users affected. Please try my guide on a ARM64 VM or device.

Thanks in advance! :)

[LMRW]

Exact same issue here too - let me know if I can help test anything!

[LMRW]

Some extra notes...

Arm64 Ubuntu 22.04.3 LTS VM (Parallels on Mac M1 host) - latest Nextcloud Snap.

$ ( /sbin/ldconfig -p || scanelf -l ) | grep fontconfig returns libfontconfig.so.1 (libc6,AArch64) => /lib/aarch64-linux-gnu/libfontconfig.so.1

I tried editing Proxy.php as mentioned in https://github.com/CollaboraOnline/richdocumentscode/issues/226#issuecomment-1750423699 but this did not work - although, I question my changes were loaded. After editing the file, how can I ask NextCloud to ensure it is using my modifications? @Jolopu

I also ensured all packages were installed... sudo apt-get update && sudo apt-get install fontconfig glibc-source -y

I see green ticks in admin confirming server is reachable, but creating and opening a new ODT file shows an infinite spinning wheel.

I am here for assistance if needed and would love to help. Thank you for the great project, but it is completely broken at the moment for every Arm64 user and has been for multiple months!... please let us help you fix it.

[LMRW]

I don't think its fair users are wasting time assuming they or their devices are at fault when in reality, arm64 simply isn't supported for last few months. Even if issue is fixed someday, I believe updating the readme until the issue is fixed is the minimum fair thing to do, to avoid users wasting hours of time today.

https://github.com/CollaboraOnline/richdocumentscode/pull/243

[LMRW]

@Jolopu

This worked for me. Edit proxy.php on lines 152 an 153:

151 exec('( /sbin/ldconfig -p || scanelf -l ) | grep fontconfig > /dev/null 2>&1', $output, $return); 152 // if ($return) 153 // return 'no_fontconfig'; 154 return '';

Fontconfig is installed and running on my server. However Collabora still thinks it's not.

Can you tell me more about how you tested this please? Maybe we can make a PR to fix the issue. I'd like to test your theory more please

[timar]

@LMRW I understand your frustration, and I'm sorry for the problems. I tried the app with nexctcloud docker on arm64 host and it worked. So it's not entirely broken.

1. On Ubuntu 20.04 docker run -d -p 80:80 --name nextcloud nextcloud
2. I installed all recommended apps
3. I installed richdocumentscode (arm64)
4. Success

Then I tried snap as suggested by @Pilzinsel64

1. On Ubuntu 20.04 sudo snap install nextcloud
2. I installed all recommended apps
3. I installed richdocumentscode (arm64)
4. Spinning spinner forever :(

root@cp-test-arm64:/var/snap/nextcloud/38464/nextcloud/extra-apps/richdocumentscode/collabora# file Collabora_Online.AppImage 
Collabora_Online.AppImage: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=30e06184968532b6a9aa36f44ada39e4af0bda56, for GNU/Linux 2.6.32, stripped

WTF?! This is unexpected. I installed the arm64 version....

Hmm, that is also there, in richdocumentscode_arm64 folder. Manually starting the AppImage lead to success...

root@cp-test-arm64:/var/snap/nextcloud/38464/nextcloud/extra-apps/richdocumentscode_arm64/collabora# ./Collabora_Online.AppImage 

Logging at warning level to file: /tmp/coolwsd.1p9A6Fu6f4/coolwsd.log
Security: coolforkit incorrect user-name, other than 'cool'
coolforkit version details: 23.05.5.3 - 5093121
Init vcl
preload: xsec_xmlsec merged ucpchelp1 wpftwriter wpftcalc wpftimpress wpftdraw writerfilter msforms ucppkg1 ucpcmis1 cached1 vbaswobj swd sw ucpdav1 smd sm pdfimport PresentationMinimizer sd scriptframe protocolhandler dlgprov date analysis vbaobj scfilt scd xmlsecurity sc expwrap oox LanguageTool ldapbe2 pcr storagefd log chartcore pdffilter migrationoo3 deploymentgui scn cui sdbt mozbootstrap bootstrap flat io animcore svgfilter embobj t602filter dbaxml stocservices chartcontroller namingservice cairocanvas invocadapt introspection migrationoo2 dbpool2 binaryurp mysql_jdbc pricing proxyfac calc invocation dba uuresolver solver reflection writer textconversiondlgs hwp msword graphicfilter emboleobj sdd slideshow dbase bib
Disabled: ucpftp1 rptxml rptui rpt dbp abp sdbc2 cmdmail PresenterScreen dbu odbc 
Allowlisted languages: de_DE el en_GB en_US es_ES fr_FR hu it nl pt_BR pt_PT ru 
Preloading dictionaries: de-DE en-US fr-FR it-IT nl-NL pt-BR ru-RU en-GB nl-BE pt-PT es-ES 
Preloading thesauri: de-DE en-US fr-FR it-IT nl-NL pt-BR ru-RU en-GB nl-BE pt-PT es-ES 
Preload icons
Preload short cut accelerators
Preload languages
Preload fonts
Preload config
office version details: { "ProductName": "Collabora Office", "ProductVersion": "23.05", "ProductExtension": ".5.3", "BuildId": "e7921fedb8b5615b0dc7cd3ec8b91c9b4e06d002", "BuildConfig": "'--enable-mpl-subset' '--with-vendor=Collabora' '--disable-community-flavor' '--with-branding=icon-themes/galaxy/brand_cp' '--with-system-dicts' '--with-myspell-dicts' '--with-system-zlib' '--disable-poppler' '--enable-cairo-rgba' '--without-system-cairo' '--without-system-fontconfig' '--without-system-freetype' '--without-system-graphite' '--without-system-harfbuzz' '--without-system-openssl' '--without-system-libpng' '--without-system-libxml' '--without-system-jpeg' '--without-system-expat' '--without-system-curl' '--without-system-icu' '--without-system-nss' '--without-system-jars' '--without-system-postgresql' '--without-java' '--without-junit' '--without-help' '--with-linker-hash-style=both' '--with-fonts' '--enable-noto-font' '--with-galleries=no' '--with-theme=colibre colibre_svg' '--with-external-thes-dir=/usr/share/mythes' '--with-external-hyph-dir=/usr/share/hyphen' '--with-external-dict-dir=/usr/share/hunspell' '--disable-dbus' '--enable-extension-integration' '--disable-odk' '--disable-kf5' '--disable-gtk3' '--disable-qt5' '--disable-gstreamer-1-0' '--disable-evolution2' '--disable-gio' '--disable-gui' '--disable-scripting-beanshell' '--disable-scripting-javascript' '--disable-ext-wiki-publisher' '--disable-report-builder' '--disable-ext-nlpsolver' '--disable-sdremote' '--disable-sdremote-bluetooth' '--disable-postgresql-sdbc' '--disable-firebird-sdbc' '--disable-randr' '--disable-ext-numbertext' '--enable-epm' '--enable-python=internal' '--disable-online-update' '--disable-dconf' '--enable-mergelibs' '--with-package-format=deb rpm' '--enable-release-build' '--with-lang=ar bg ca cs da de el en-US en-GB eo es eu fi fr gl he hr hu id is it ja ko lo nb nl oc pl pt pt-BR sq ru sk sl sv tr uk vi zh-CN zh-TW' '--disable-lotuswordpro' '--disable-lpsolve' '--enable-symbols' '--enable-sal-log' '--without-templates' '--with-external-tar=/home/collabora/jenkins/external' '--disable-symbols' '--with-package-format=deb' '--srcdir=/home/collabora/jenkins/workspace/core-co-22.05-for-arm64-PI' '--enable-option-checking=fatal'" }
Ready to accept connections on port 9983.

Something may be wrong with the nextcloud snap.

[Pilzinsel64]

@timar Thank you very much for trying thisa out!

Well, the result is unexpected for me. 🤔 The only idea I have could be a missing or wrong dependency. What exactly has changed in-between this both version (23.5.5 and 22.05.1301) regarding dependencies?

Edit: I just see that it tries to start the x86_64 version and not the arm64 version.Do you have installed both? (Maybe "recommended apps" installed the non-arm64 version)