Preliminary work for Seafile 10.0

[ChatDeBlofeld]

Hi folks,

I've started the work to build Seafile 10. Compilation dependencies were straightforward to update (see ChatDeBlofeld/seafile-rpi@321057d8febdf4b75ccafce8c9a9167f034e5a02) but python packages have been completely reworked upstream and I don't know how to merge them with the custom requirements we have here (which still have pending changes #120)

What I can tell so far:

• #113 is still needed, no update here.
• Cryptography downgrade (#99) obviously still needed for Buster.
• Golang minimal version is 1.17 since haiwen/seafile-server@df78d10ec15d8d2b570d531f4cad835c7e874535, which is higher than ones provided in Debian Buster (1.11), Bullseye (1.15) and Ubuntu 20.04 (1.13). Honestly, I don't think it would be that bad not to build the fileserver when the go version is too old...

By disabling the go fileserver when needed, I was able to build something usable for:

• Debian Buster
• Debian Bullseye
• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 22.10

Tested features (22.04 only):

• Web UI (upload/download)
• Sync
• Webdav
• Memcached
• sqlite backend
• MariaDB backend
• MySQL backend

[lanmarc77]

Thank you for maintaining this here. Not really understanding a lot of the seafile compiling process, might I throw into the ring the idea to build the go part on e.g. a docker based environment and then statically link the output of the compiler? Doesn't the go part generate binaries anyways? This way general usage binaries could be created which not even rely on a locally libc of a specific version or OS. Additionally it might also be easy to generate arch64 binaries.

[Duke690]

I simply upgraded the golang version for my raspberrypi (bullseye) with this tutorial https://www.e-tinkers.com/2019/06/better-way-to-install-golang-go-on-raspberry-pi/

I was then able to build seafile 10.0.1 with the build.sh script. I had to remove the line in build.sh where golang is installed via apt. I used the same requirements as for v9.0.9. for seafdav and the ones ChatDeBlofeld mentioned for seahub (https://github.com/haiwen/seahub/blob/master/requirements.txt)

I second lanmarc77's suggestion to use a docker based build environment.

[nihilistaX]

I was able to build seafile 10.0.1, golang is not a problem as you can install and activate every version with gvm. So build runs fine, but after installing i got internal server error. Seems something went wrong with seahub.

[Duke690]

I was able to build seafile 10.0.1, golang is not a problem as you can install and activate every version with gvm. So build runs fine, but after installing i got internal server error. Seems something went wrong with seahub.

Where do you get that internal server error? I got an error starting seahub when using the v9.0.9 requirements to build. But after using the ones mentioned above my installation works flawless so far.

[nihilistaX]

Seahub starts without error, but web interface shows internal error. Used same requirements as above. And stopping seahub shows "failed to stop seahub", but it is stopped.....Maybe some problem with some requirement or somewhat.

[ChatDeBlofeld]

@nihilistaX Try to set deamon = False in gunicorn.conf.py. If exceptions arise in Seahub, stack trace will be displayed.

@lanmarc77 I won't work on such an integration but yes it certainly works. Universal binaries will be harder though since the server depends on libc (see below), I don't even know if it possible to static link it... However, I built a fileserver on buster (which has oldest versions of dependencies) which should work fine on later versions.

$ ldd fileserver 
        linux-vdso.so.1 (0x0000007f8b365000)
        libresolv.so.2 => /lib/aarch64-linux-gnu/libresolv.so.2 (0x0000007f8b30e000)
        libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000007f8b2fa000)
        libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000007f8b2c9000)
        libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000007f8b154000)
        /lib/ld-linux-aarch64.so.1 (0x0000007f8b335000)

---

I've added support for notification server (https://github.com/ChatDeBlofeld/seafile-rpi/commit/23acec907978fee8e074b023feec6559a63b7873), which is written in go too 🙃. Pretty ugly, backward compatibility is broken (but it was already the case when the go fileserver was added and it seemed ok at the time) and I patch the build-server.py script on the fly to copy the new server properly. I will open a PR (https://github.com/haiwen/seahub/pull/5453) to try to improve the later.

[Gustl22]

@ChatDeBlofeld thanks for your work! I wrote the maintainers to merge your PR (but don't know, if they actually do). Also if you feel ready you can start a PR for v10. I also wrote you a message on LinkedIn.

[ChatDeBlofeld]

PR was merged, thanks!

I will start a PR for 10.0.1, maybe should we write down somewhere the minimal go requirements to build the server?

[ChatDeBlofeld]

Hi @lanmarc77 @Duke690 @nihilistaX

Static linking libc in go happened to be very easy, so I set up an experimental docker pipeline to cross-compile seafile from amd64. You'll find some compiled packages for seafile 10 here. If some of you want to test them it would be very helpful!

[lanmarc77]

Hi @ChatDeBlofeld , @Gustl22 thank you very much for your work. I integrated the docker pipeline with github actions. This way packages can be built online. You find the the finally complete build here: https://github.com/lanmarc77/seafile-rpi/actions/runs/5031423956/jobs/9024494684 I takes around 1hr per distribution and fits very well into the free 2000 github actions minutes per month.

I would like to add automated tests of these builds to get rid of the human more or less completely for the testing process. So I am thinking of starting a container installing the build package in it probably with different database backends and then test if all the processes start and keep running. Are there more sophisticated tests needed?

And how about renaming seafile-rpi into seafile-arm? All the best and thank you again.

[lanmarc77]

Small Update, went down the rabbit hole kinda...

I am now using the newest Go version 1.20.4 which is downloaded and installed inside the build container during creation. This allows compiling also for older distributions directly in the container type it is meant to run later. Except for focal arm/v7 everything can be build and compiled using Github actions. Focal arm/v7 I think is a bug inside Github actions itself as it does compile locally just not online with the same script.

I am currently compiling only with the requirement that come with the original repos. Are there any reasons beside maybe historical ones that there are version specific requirements which you apply ahead? It did compile without them, also for tests I did with 9.0.9 and 8.0.7.

I released 10.0.1 for all major distributions for everyone who wants to test: https://github.com/lanmarc77/seafile-arm/releases

No tests have been conducted so far.

[Gustl22]

Wow @ChatDeBlofeld @lanmarc77 that looks very promising, if you feel comfortable, you can start a Merge Request, if you want, or even start it as Draft, if you don't feel ready to merge yet ;D

[lanmarc77]

Hi @Gustl22 , I am unsure before I could implement automated tests. Can you help with some of my questions?

[nihilistaX]

Hi, tested bullseye aarch64 and armv7l builds, as upgrade and as fresh install, but seahub won't start. That's what i got on fresh aarch64 install: Starting seahub at port 8000 ... Traceback (most recent call last): File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/bin/gunicorn", line 8, in <module> sys.exit(run()) File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/app/wsgiapp.py", line 67, in run WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/app/base.py", line 231, in run super().run() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/app/base.py", line 72, in run Arbiter(self).run() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/arbiter.py", line 58, in __init__ self.setup(app) File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/arbiter.py", line 118, in setup self.app.wsgi() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/app/base.py", line 67, in wsgi self.callable = self.load() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/app/wsgiapp.py", line 58, in load return self.load_wsgiapp() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp return util.import_app(self.app_uri) File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/util.py", line 359, in import_app mod = importlib.import_module(module) File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1030, in _gcd_import File "<frozen importlib._bootstrap>", line 1007, in _find_and_load File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 680, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 790, in exec_module File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed File "/opt/seafile/seafile-server-10.0.1/seahub/seahub/wsgi.py", line 25, in <module> application = get_wsgi_application() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/django/core/wsgi.py", line 12, in get_wsgi_application django.setup(set_prefix=False) File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/django/__init__.py", line 24, in setup apps.populate(settings.INSTALLED_APPS) File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/django/apps/registry.py", line 122, in populate app_config.ready() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/statici18n/apps.py", line 8, in ready from . import conf # noqa File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/statici18n/conf.py", line 3, in <module> from appconf import AppConf ImportError: cannot import name 'AppConf' from 'appconf' (/usr/local/lib/python3.9/dist-packages/appconf/__init__.py) Error:Seahub failed to start.

[abutre]

Hi, tested bullseye aarch64 and armv7l builds, as upgrade and as fresh install, but seahub won't start. That's what i got on fresh aarch64 install: Starting seahub at port 8000 ... Traceback (most recent call last): File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/bin/gunicorn", line 8, in <module> sys.exit(run()) File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/app/wsgiapp.py", line 67, in run WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/app/base.py", line 231, in run super().run() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/app/base.py", line 72, in run Arbiter(self).run() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/arbiter.py", line 58, in __init__ self.setup(app) File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/arbiter.py", line 118, in setup self.app.wsgi() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/app/base.py", line 67, in wsgi self.callable = self.load() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/app/wsgiapp.py", line 58, in load return self.load_wsgiapp() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp return util.import_app(self.app_uri) File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/gunicorn/util.py", line 359, in import_app mod = importlib.import_module(module) File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1030, in _gcd_import File "<frozen importlib._bootstrap>", line 1007, in _find_and_load File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 680, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 790, in exec_module File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed File "/opt/seafile/seafile-server-10.0.1/seahub/seahub/wsgi.py", line 25, in <module> application = get_wsgi_application() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/django/core/wsgi.py", line 12, in get_wsgi_application django.setup(set_prefix=False) File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/django/__init__.py", line 24, in setup apps.populate(settings.INSTALLED_APPS) File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/django/apps/registry.py", line 122, in populate app_config.ready() File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/statici18n/apps.py", line 8, in ready from . import conf # noqa File "/opt/seafile/seafile-server-10.0.1/seahub/thirdpart/statici18n/conf.py", line 3, in <module> from appconf import AppConf ImportError: cannot import name 'AppConf' from 'appconf' (/usr/local/lib/python3.9/dist-packages/appconf/__init__.py) Error:Seahub failed to start.

Solved for me in Ubuntu by installing python3-django-appconf

[lanmarc77]

@nihilistaX , yes I already detected missing dependencies in the packaging and adjusted the new build scripts. ArmV7 is not building online but arm64 is. I was also able to implement an own test circle that installs the build package into a vanilla container and tries to let it run with sqlite and mysql backend and then checks if all the services come up and if the initial login page is reachable. Can you try the following auto build package for bullseye/arm64 if this runs smoothly https://github.com/lanmarc77/seafile-arm/suites/13220402545/artifacts/720215176 ? I would almost bet it does as it passed the automatic test as well. @abutre Installing the missing dependencies manually almost always fixes module not found errors. Still I try to built packages that come with everything bundled so that only the db and python3 need to be installed.

[lanmarc77]

@ChatDeBlofeld, @Gustl22 In addition to downloading the newest go instead of using the distribution one now also the newest rust is downloaded and installed. This allows compiling of the newest python cryptography library as well so that we can get more or less CVE free libraries even on older distros.

[nihilistaX]

@nihilistaX , yes I already detected missing dependencies in the packaging and adjusted the new build scripts. ArmV7 is not building online but arm64 is. I was also able to implement an own test circle that installs the build package into a vanilla container and tries to let it run with sqlite and mysql backend and then checks if all the services come up and if the initial login page is reachable. Can you try the following auto build package for bullseye/arm64 if this runs smoothly https://github.com/lanmarc77/seafile-arm/suites/13220402545/artifacts/720215176 ? I would almost bet it does as it passed the automatic test as well. @abutre Installing the missing dependencies manually almost always fixes module not found errors. Still I try to built packages that come with everything bundled so that only the db and python3 need to be installed.

Will test that maybe tomorrow....

Edit: Now seahub starts. No time for more testing, but seems to work.

[lanmarc77]

@nihilistaX , thank you for your time. Meanwhile I updated all other packages and hope for anyone who can check and report back https://github.com/lanmarc77/seafile-arm/releases Ideally this is the last human test as the automatic tests can take over after they proven to be enough.

[nihilistaX]

@lanmarc77 Installed seafile-server-10.0.1-buster-armv7l on another device and bullseye-aarch64 on my maain device. Seems to work....but as before i didn't test all functions. Did a little upload, thats working.

[lanmarc77]

I was able to update my own system from version 9 using bullseye armv7 and everything I use works without issues.

[ChatDeBlofeld]

Hi @lanmarc77, that's a great work!

Here some thoughts, you may find something informative.

Are there more sophisticated tests needed?

Most of the time when something doesn't work it is due to missing dependencies, other cases are probably too hard to forecast and don't worth the effort. Then I think just checking if it launches is enough.

I am now using the newest Go version 1.20.4 which is downloaded and installed inside the build container during creation.

In this case you don't need to link statically the go parts but it probably doesn't hurt. I've deliberately avoided to install Go packages because I target RISCV too and the packages don't exist, but that's my business. :)

This allows compiling also for older distributions directly in the container type it is meant to run later.

Just warning, but maintaining a per-distro pipeline can be cumbersome, qemu and docker are sometimes...capricious. For example I can't build for focal targeting arm/v7 because of this. Just ran a build for jammy and got some TLS handshake errors, didn't dig further. It can be tiring.

As you have seen, I'm not often implied. That's why I tried to make a "universal" package based on buster to reduce maintenance. It implies to ship less python dependencies though (mostly every that are not pure python) but nothing not written in the manual. In a way, I find it very convenient because building a wheel for cryptography is the most painful thing I've experienced in this project, with weird stupid bugs arising from time to time (and I didn't even know there were CVE on top of that).

Are there any reasons beside maybe historical ones that there are version specific requirements which you apply ahead?

Not sure, but requirements used to be regularly broken. Since version 10 most versions are pinned, so it should be better.

[lanmarc77]

Hi @ChatDeBlofeld, thank you for your input and the historic facts. The problems building for arm/v7 only happen online for me. Using my Arch based build system I do not see any issues at all. But I understand from your experience that this also might change in the future. So another option (instead of building distro based) could be one universal package that only contains specific python packages. I guess this would need the users then to install the missing packages themselves and we would need to make sure that they exist in the users distro (or via requirements) and that they are compatible with our universal package. Would this not bring is into dependency issues on the users side that then end up here or in the forums and also result in (support/debugging) work? I also read that e.g. using pyinstaller python can generate executables that then include python itself. Maybe this could also be another way?

[ChatDeBlofeld]

I guess this would need the users then to install the missing packages themselves and we would need to make sure that they exist in the users distro (or via requirements) and that they are compatible with our universal package.

Indeed, but the work should be done for us by the devs, as this is the way they ship the amd64 packages. The manual specifies a bunch of prerequisites which work for us too. But I agree this is less convenient for the users.

Would this not bring is into dependency issues on the users side that then end up here or in the forums and also result in (support/debugging) work?

Probably. These issues should intersect with ones from amd64 users though, then the trade-off looks ok to me.

I also read that e.g. using pyinstaller python can generate executables that then include python itself. Maybe this could also be another way?

Wouldn't this slope eventually lead us to some containerization tool like docker? :D Not that I wouldn't like it, for me it is the best way to deploy such an application, but not sure it's the goal here (or it is?).

Anyway, there is pros and cons in both case...

[lanmarc77]

Indeed, we finally would end up with docker. I just checked, there seems to be no armv7 version though. Yet...

My next steps are cleanup and documentation of how the build process works and then stay with my fork and wait for the next update to see how much this works with new releases. Ideally no intervention is needed. Even more ideal would be if the original project checks the build process and uses the tests before actually making a new release. I will try to reach out to them after cleanup and documentation.

Thank you for your input.

[nihilistaX]

After some time using this builds on 2 machines i see that seafdav seems not to work. Checked seafdav port but thats not opened. Found that in controller.log: 2023-06-14 22:04:04 seafile-controller.c(124): Traceback (most recent call last): File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/root/seafile/seafile-server-10.0.1/seahub/thirdpart/wsgidav/server/server_cli.py", line 48, in <module> from wsgidav.default_conf import DEFAULT_CONFIG, DEFAULT_VERBOSE File "/root/seafile/seafile-server-10.0.1/seahub/thirdpart/wsgidav/default_conf.py", line 18, in <module> from wsgidav.dir_browser import WsgiDavDirBrowser File "/root/seafile/seafile-server-10.0.1/seahub/thirdpart/wsgidav/dir_browser/__init__.py", line 2, in <module> from ._dir_browser import WsgiDavDirBrowser File "/root/seafile/seafile-server-10.0.1/seahub/thirdpart/wsgidav/dir_browser/_dir_browser.py", line 13, in <module> from jinja2 import Environment, FileSystemLoader, select_autoescape File "/root/seafile/seafile-server-10.0.1/seahub/thirdpart/jinja2/__init__.py", line 12, in <module> 03Q2023-06-14 22:04:04 seafile-controller.c(124): from .environment import Environment File "/root/seafile/seafile-server-10.0.1/seahub/thirdpart/jinja2/environment.py", line 25, in <module> from .defaults import BLOCK_END_STRING File "/root/seafile/seafile-server-10.0.1/seahub/thirdpart/jinja2/defaults.py", line 3, in <module> from .filters import FILTERS as DEFAULT_FILTERS # noqa: F401 File "/root/seafile/seafile-server-10.0.1/seahub/thirdpart/jinja2/filters.py", line 13, in <module> from markupsafe import soft_unicode ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/root/seafile/seafile-server-10.0.1/seahub/thirdpart/markupsafe/__init__.py) ser/__init__.py", line 2, in <module> from ._dir_browser import WsgiDavDirBrowser File "/root/seafile/seafile-server-10.0.1/seahub/thirdpart/wsgidav/dir_browser/_dir_browser.py", line 13, in <module> from jinja2 import Environment, FileSystemLoader, select_autoescape File "/root/seafile/seafile-server-10.0.1/seahub/thirdpart/jinja2/__init__.py", line 12, in <module> 03Q

[ChatDeBlofeld]

This is caused by https://github.com/haiwen/seafile-rpi/issues/113

Forgot it when saying upstream requirements used to be broken, they still are...

[lanmarc77]

Hi, I updated the build system with markupsafe fixed to the version needed by seafdav and also included automated tests to check if seafdav also starts. As I am not using this webdav bridge I was not yet aware of this. All files in my release were updated accordingly. I also checked the docker release and it does not contain this markupsafe in the thirdparty folder. I will put this on my list to ask if this could be added to the requirements upstream after my cleanup. If @nihilistaX could check if seafdav works as expected that would be great. Thanks @ChatDeBlofeld to point me in the right direction with the issue.

[nihilistaX]

First short test on armv7l and aarch64: Seems to work now. Couldnt get inside subfolders first but maybe thats maybe cache issue. With /seafdav/subfolder its works. After manually typing that all fine for now. Edit: Strange, in browser links go to /xyz instead of /seafdav/xyz....with file manager on phone all fine.....btw

[Gustl22]

Thank you all @ChatDeBlofeld @lanmarc77 @nihilistaX . Sorry that I stood out of the conversation. As I'm not using the cloud on arm anymore on a regular basis, I'm not familiar with the current development of seafile, and also I cannot put too much time into it. If you come up with a working solution, I'm happy to review any Merge Requests and then we can push the active development with further volunteers. Thank you!