syncing-server-js
syncing-server-js copied to clipboard
standardnotes
ARM64 version (for Raspberry Pi) ?
I've set up the environment files and ran the startup file, but it seems like there's an issue with MySQL that prevents it from working.
myusername@raspberrypi:~/standalone $ ./server.sh start
Starting up infrastructure
# ...
Step 14/14 : CMD [ "start-web" ]
# ...
Status: Downloaded newer image for standardnotes/auth:dev
Pulling db (mysql:5.6)...
5.6: Pulling from library/mysql
ERROR: no matching manifest for linux/arm64/v8 in the manifest list entries
The same error also occurs for the setup script in the "standalone" repo.
From a quick search, the Docker images for MySQL doesn't seem to have any ARM64 versions, and I think there was some workaround with MariaDB in the deprecated syncing server for this, but I'm unsure how to do that.
Would be great if there is a way to get it running on a Raspberry Pi. Thank you!
Machine info
myusername@raspberrypi:~/syncing-server-js $ hostnamectl
Static hostname: raspberrypi
Icon name: computer
Machine ID: XXXX
Boot ID: XXXX
Operating System: Debian GNU/Linux 10 (buster)
Kernel: Linux 5.10.17-v8+
Architecture: arm64
The OS I installed is 64-bit version of the Raspberry Pi OS, from here.
Change mysql:5.6 to mariadb:10.5 in docker-compose.yml
The next problem is half the other images for standardnotes are not supported by ARM so it still doesn't work.
Nice one @trinsic
Let's try to fix them one at a time.
I was able to successfully build the Dockerfile on multiple architectures using docker buildx. (I will test them on my RPI in the upcoming weeks. IMO it should run just fine as it is nodejs)
@JaspalSuri Is it possible to integrate this build process in the upstream and create multi-arch Docker images? At least for this repo?
docker buildx build --platform linux/arm64,linux/arm/v7,linux/arm/v6,linux/amd64 .
EDIT: I will post the aggregated docker images soon.
Yeah, so I only have a Pi 3B with 1GB ram and the only one I couldn't build was auth (OOM). Hopefully, someone can get a multi-arch running, that would be super.
It seems that you are building on the RPi. You can use docker buildx to build on a *nix laptop. It probably uses some qemu magic under the hood to compile for multiple architectures. Anyways let me try to compile it in my free time.
Here are the multi-arch builds until we have official ones.
blmhemu/snotes-syncing-server-js:latest
blmhemu/snotes-api-gateway:latest
blmhemu/snotes-auth:latest
Try them out and let me know if they work !
Yeah, it works just fine. Changed to MariaDB due to missing arm64 MySQL image.
Happy to have it running on my Pi.
Thank you for creating the new images! I would very much like to install Standard Notes on my RPI4 but running a 32bit arm os. Replacing mysql with maridb does not work, although I have tried a lot. Do you have any clues what could help? Thanks a lot!
Here is the error message:
2021-08-28 19:23:46 0 [ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded,
2021-08-28 19:24:18 0 [ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded,
2021-08-28 19:24:18 0 [Warning] 'default-authentication-plugin' is MySQL 5.6 / 5.7 compatible option. To be implemented in later versions.,
2021-08-28 19:24:18 0 [Note] InnoDB: Buffer pool(s) load completed at 210828 19:24:18,
2021-08-28 19:24:18 1 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1017: Can't find file: './mysql/' (errno: 2 "No such file or directory"),
2021-08-28 19:24:18 0 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist,
2021-08-28 19:24:18 0 [Note] Server socket created on IP: '::'.,
2021-08-28 19:24:18 0 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.db' doesn't exist,
2021-08-28 19:24:18 0 [ERROR] Aborting
This is the corresponding part of my docker-compose.yml:
image: linuxserver/mariadb
container_name: db
environment:
MYSQL_DATABASE: '${DB_DATABASE}'
MYSQL_USER: '${DB_USERNAME}'
MYSQL_PASSWORD: '${DB_PASSWORD}'
MYSQL_ROOT_PASSWORD: '${DB_PASSWORD}'
expose:
- 3306
restart: unless-stopped
command: mysqld --default-authentication-plugin=mysql_native_password --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
volumes:
- ./data/mysql:/var/lib/mysql
- ./data/import:/docker-entrypoint-initdb.d
networks:
- standardnotes_standalone
I just created a working Standalone installation for the Raspberry Pi.
Here is the link to the repo: https://github.com/agmm/standalone-rpi
Thank you @blmhemu for the images! They work fine on my RPi4.
Am I correct in assuming though, that currently there's nobody maintaining these arm64 compatible images to stay up to date with the latest releases? Meaning I can't update my standard notes server or any clients (Desktop, Mobile) due to compatibility issues, unless someone builds new server images and makes them available on Docker Hub? Do you intent to update your images once in a while @blmhemu?
Is there a guide somewhere that goes over building my own arm64 images and using them in my docker-compose.yml?
Hey ! I have requested the maintainers to integrate the build script, (but there has not been any progress ?). For, updates, you can build the images using docker buildx.
docker buildx build --platform linux/arm64,linux/arm/v7,linux/arm/v6,linux/amd64 .
The Dockerfile itself is unchanged if I remember correctly.
It doesn't seem to work as flawlessly for me as it might have worked for you :)
When running your command to build the arm versions, I run into
- an error about node not getting found
- an error about python not getting found
I can fix those by symlinking to the correct node path and by setting the python path in the dockerfile
RUN sh -c 'mkdir /usr/local/sbin && ln -s /usr/local/bin/node /usr/local/sbin/node;
ENV PYTHON="/usr/bin/python3"
However, I still get a bunch of errors from node-gyp. It's the same behavior/errors for any of the three components: syncing server, auth and api-gateway.
Any idea?
> [ 8/10] RUN yarn install --pure-lockfile:
#0 5.823 yarn install v1.22.15
#0 8.075 [1/5] Validating package.json...
#0 8.165 [2/5] Resolving packages...
#0 17.11 [3/5] Fetching packages...
#0 197.6 info [email protected]: The platform "linux" is incompatible with this module.
#0 197.6 info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
#0 197.8 info [email protected]: The platform "linux" is incompatible with this module.
#0 197.8 info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
#0 197.9 [4/5] Linking dependencies...
#0 252.7 [5/5] Building fresh packages...
#0 262.8 error /var/www/node_modules/microtime: Command failed.
#0 262.8 Exit code: 1
#0 262.8 Command: node-gyp-build
#0 262.8 Arguments:
#0 262.8 Directory: /var/www/node_modules/microtime
#0 262.8 Output:
#0 262.8 Error while loading /tmp/yarn--1647984361119-0.4700224287790489/node-gyp: No such file or directory
#0 262.8 info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
#0 298.9 warning Error running install script for optional dependency: "/var/www/node_modules/@standardnotes/domain-events-infra/node_modules/@newrelic/native-metrics: Command failed.
#0 298.9 Exit code: 1
#0 298.9 Command: node ./lib/pre-build.js install native_metrics
#0 298.9 Arguments:
#0 298.9 Directory: /var/www/node_modules/@standardnotes/domain-events-infra/node_modules/@newrelic/native-metrics
#0 298.9 Output:
#0 298.9 ============================================================================
#0 298.9 Attempting install in native-metrics module. Please note that this is an
#0 298.9 OPTIONAL dependency, and any resultant errors in this process will not
#0 298.9 affect the general performance of the New Relic agent, but event loop and
#0 298.9 garbage collection metrics will not be collected.
#0 298.9 ============================================================================
#0 298.9
#0 298.9 > /usr/local/bin/node /usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js clean configure
#0 298.9 gyp info it worked if it ends with ok
#0 298.9 gyp info using [email protected]
#0 298.9 gyp info using [email protected] | linux | arm64
#0 298.9 gyp info find Python using Python version 3.9.7 found at \"/usr/bin/python3\"
#0 298.9 gyp http GET https://nodejs.org/download/release/v16.13.1/node-v16.13.1-headers.tar.gz
#0 298.9 gyp http 200 https://nodejs.org/download/release/v16.13.1/node-v16.13.1-headers.tar.gz
#0 298.9 gyp http GET https://nodejs.org/download/release/v16.13.1/SHASUMS256.txt
#0 298.9 gyp http 200 https://nodejs.org/download/release/v16.13.1/SHASUMS256.txt
#0 298.9 gyp info spawn /usr/bin/python3
#0 298.9 gyp info spawn args [
#0 298.9 gyp info spawn args '/usr/local/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py',
#0 298.9 gyp info spawn args 'binding.gyp',
#0 298.9 gyp info spawn args '-f',
#0 298.9 gyp info spawn args 'make',
#0 298.9 gyp info spawn args '-I',
#0 298.9 gyp info spawn args '/var/www/node_modules/@standardnotes/domain-events-infra/node_modules/@newrelic/native-metrics/build/config.gypi',
#0 298.9 gyp info spawn args '-I',
#0 298.9 gyp info spawn args '/usr/local/lib/node_modules/npm/node_modules/node-gyp/addon.gypi',
#0 298.9 gyp info spawn args '-I',
#0 298.9 gyp info spawn args '/home/syncingserver/.cache/node-gyp/16.13.1/include/node/common.gypi',
#0 298.9 gyp info spawn args '-Dlibrary=shared_library',
#0 298.9 gyp info spawn args '-Dvisibility=default',
#0 298.9 gyp info spawn args '-Dnode_root_dir=/home/syncingserver/.cache/node-gyp/16.13.1',
#0 298.9 gyp info spawn args '-Dnode_gyp_dir=/usr/local/lib/node_modules/npm/node_modules/node-gyp',
#0 298.9 gyp info spawn args '-Dnode_lib_file=/home/syncingserver/.cache/node-gyp/16.13.1/<(target_arch)/node.lib',
#0 298.9 gyp info spawn args '-Dmodule_root_dir=/var/www/node_modules/@standardnotes/domain-events-infra/node_modules/@newrelic/native-metrics',
#0 298.9 gyp info spawn args '-Dnode_engine=v8',
#0 298.9 gyp info spawn args '--depth=.',
#0 298.9 gyp info spawn args '--no-parallel',
#0 298.9 gyp info spawn args '--generator-output',
#0 298.9 gyp info spawn args 'build',
#0 298.9 gyp info spawn args '-Goutput_dir=.'
#0 298.9 gyp info spawn args ]
#0 298.9 gyp info ok
#0 298.9 > /usr/local/bin/node /usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js build -j 1 native_metrics
#0 298.9 gyp info it worked if it ends with ok
#0 298.9 gyp info using [email protected]
#0 298.9 gyp info using [email protected] | linux | arm64
#0 298.9 gypError while loading /tmp/yarn--1647984361119-0.4700224287790489/make: No such file or directory
#0 298.9 info spawn make
#0 298.9 gyp info spawn args [ 'native_metrics', 'BUILDTYPE=Release', '-C', 'build', '--jobs', 1 ]
#0 298.9 gyp ERR! build error
#0 298.9 gyp ERR! stack Error: `make` failed with exit code: 1
#0 298.9 gyp ERR! stack at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:194:23)
#0 298.9 gyp ERR! stack at ChildProcess.emit (node:events:390:28)
#0 298.9 gyp ERR! stack at Process.ChildProcess._handle.onexit (node:internal/child_process:290:12)
#0 298.9 gyp ERR! System Linux 5.10.0-12-amd64
#0 298.9 gyp ERR! command \"/usr/local/bin/node\" \"/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js\" \"build\" \"-j\" \"1\" \"native_metrics\"
#0 298.9 gyp ERR! cwd /var/www/node_modules/@standardnotes/domain-events-infra/node_modules/@newrelic/native-metrics
#0 298.9 gyp ERR! node -v v16.13.1
#0 298.9 gyp ERR! node-gyp -v v8.3.0
#0 298.9 gyp ERR! not ok
#0 298.9 Failed to execute native-metrics install: No pre-built artifacts for your OS/architecture."
#0 298.9 info This module is OPTIONAL, you can safely ignore this error
------
Dockerfile:24
--------------------
22 | COPY --chown=$UID:$GID package.json yarn.lock /var/www/
23 |
24 | >>> RUN yarn install --pure-lockfile
25 |
26 | COPY --chown=$UID:$GID . /var/www
--------------------
error: failed to solve: process "/dev/.buildkit_qemu_emulator /bin/sh -c yarn install --pure-lockfile" did not complete successfully: exit code: 1
Ok, I've figured it out. The problem seems to be with trying to do a multi-arch build with the command proposed by blmhemu.
docker buildx build --platform linux/arm64,linux/arm/v7,linux/arm/v6,linux/amd64 .
When executing the build command for the first time, I get an error message:
error: docker exporter does not currently support exporting manifest lists
The "solution" to that issue is to create a custom builder with docker buildx create, however, this leads to lots of build errors as shown above. I deleted the custom builder again (docker buildx rm) and was able to successfully do arm builds by only building a single architecture without any code changes to the official SN repos.
docker buildx build --platform linux/arm64 --load .
I don't know if the code has changed since blmhemu built the images or if some special setup for buildx is needed to be able to create multi-arch images. 🤷
There might be some buildx dark magic which I may have not understood fully.