sandstorm
sandstorm copied to clipboard
sandstorm-io
Arm Architecture (Raspberry Pi) - Would it be possible ?
Hi everyone !
A lot of people are adopting the Raspberry Pi as a solution for self hosting. I understand (if I get things right) that you guys had to choose an architecture once in for all for compiling the sandstorm application server and make it easy for users to download and install already compiled apps. I understand enabling a new architecture means compiling every version of sandstorm and apps that goes available (just like we do for linux distros and packages). However, taking into account the growing "market share" (if you allow me) of arm platforms, I think it would be a good idea. Of course, I'm willing to gather informations about your process/guidelines when it comes to compilation and release and to do this task, if you accept.
It would be great to have a sandstorm instance running on a Raspberry Pi 3 (for instance). Here are the specs from www.raspberrypi.org
Specifications SoC: Broadcom BCM2837 CPU: 4× ARM Cortex-A53, 1.2GHz GPU: Broadcom VideoCore IV RAM: 1GB LPDDR2 (900 MHz) Networking: 10/100 Ethernet, 2.4GHz 802.11n wireless Bluetooth: Bluetooth 4.1 Classic, Bluetooth Low Energy Storage: microSD GPIO: 40-pin header, populated Ports: HDMI, 3.5mm analogue audio-video jack, 4× USB 2.0, Ethernet, Camera Serial Interface (CSI), Display Serial Interface (DSI)
I am still kinda surprised that this request comes up so much, given that 1 GB RAM spec. I don't know how much the OS and Sandstorm itself would use, but running Sandstorm grains average about 100 MB of RAM each. So, I suppose it'd theoretically work, but resources would be incredibly constrained.
I also think earlier on MongoDB didn't work on ARM, which Sandstorm requires. I heard they got it working, allegedly, since.
Given the other complications you mentioned above, note that Sandstorm is still very early in development, and it probably makes sense to get it stable on one architecture before adding more.
@paulproteus: IMHO, there should be an FAQ question somewhere for "why doesn't Sandstorm currently support ARM?". It gets asked a lot. By definition, a frequently asked question.
@kentonv from about a month ago: " Sandstorm will support ARM someday but it's going to require a large investment in tooling in order to be painless for developers."
@ocdtrekkie did a pretty good job of explaining it already, but to summarize the technical reasons we don't plan to support ARM in the near future:
- MongoDB (which Meteor and Sandstorm rely on) doesn't officially support ARM yet: https://jira.mongodb.org/browse/SERVER-1811
- Since Sandstorm packages contain native code, either app developers would need to build packages for every platform supported, or we'd need to have some way to automate building these packages, lest we wind up with significant app ecosystem fragmentation. Building that tooling would take time and attention from other things which we think will better serve a greater number of users right now.
As a result of these factors, supporting Sandstorm on ARM is not a priority for us right now, though I agree it's a thing we want to work in the longer future.
Hi @memento ,
I wrote up some docs in #2432 about the status quo.
Are you still possibly interested in working on this? If so, then I'm happy to discuss details.
Let me know what you think!
Just some observations:
- 1 GiB of memory in an arm system is actually pretty easy to come by these days.
- Memory usage would probably be a bit lower anyway on a 32-bit platform.
Err.. I may have misunderstood what was being said about the ram; sorry for the noise :/
For some environments 1gb of RAM (on RPIs) could be enough. Everyone should agree that there is no such thing as running massive databases doing big data analysis or comparable operations on RPIs, but personally I see some usage in having container orchestration on these small ARM boards; e.g. I run container for all my little nifty home automatization scripts / tools on my RPIs. In fact, it shouldn't be that amount of work to optimize or strip down sandstorm for the arm64 architecture to keep it lightweight.
@memento : let's fork sandstorm to arm64 and just leave the building of compatible arm64 sandstorm apps to the current app developers.
A fork seems premature; the developers haven't said they're closed to the idea, just that it isn't a priority for them. @paulproteus expressed openness to it.
It would be good to enumerate what actually needs to be done to get sandstorm to work on non x86_64 besides re-compiling. The mongodb issue is apparently solved. Here's what else I can think of, not being terribly familiar with the code:
- I suspect the seccomp-bpf rules would have to be extended for each arch, since rules match per-arch. I imagine this is a fairly mechanical change though.
- Per @zarvox's second point, we'll need to figure out how to deal with the possibility of native code for different CPUs. I'm sure this is soluable, but is going to require a bit of design work. I suspect the most challenging part of this is to find a way to make it easy for developers to build apps for different architectures.
Any other things people know of that would have to change?
//cc @kentonv
Re (2), perhaps worth looking at how various linux distros deal with this; I know debian has a fair amount of machinery, though I'm not intimately familiar with them.
@paulproteus @3mbr4c3
Hey guys. Still interrested in that plaform (a lot). I'm currently deep into freelancing right now. As soon as I earn enough to finish the year (the sinews of war) I'ld like to participate on this task (a few months I guess).
Chears,
FWIW, people often find that 1GB of RAM is not enough for Sandstorm. It's pretty RAM-hungry since for some reason everyone wants to write apps in bulky garbage-collected dynamic languages like Ruby or Java. :)
But yeah, I'm happy to merge patches that make Sandstorm build on ARM if you want to play around with it.
I think it's pretty clear how we'd extend the low-level package format to support multi-arch. Take a look at the Archive type at the bottom of package.capnp -- that's the actual package format. We could add a way for a file to have multiple versions targeting different archs, and then the unpacker would only extract the one for the target arch.
The hard part, though, is providing tools to the developers such that they're able to construct these packages. The current design of the tooling isn't exactly well-designed for this, since it sort of slurps up binaries off the host system. If we designed tools where the user specifies a more precise package list (e.g. from Debian or maybe Nix) then it would be a lot easier to grab the packages for multiple archs. I think work on such tooling could be a pretty interesting project that someone could easily own and work on without requiring lots of coordination with other parts of the project, if someone is interested!
@memento: just get in contact, I'll open my SandstormArm64 repo this weekend.
I currently ported just 20% of core sandstorm to arm64, it will take some more time before I / we can take a look into the toolchain. As stated before, I will open my repo which I just forgot right now, but I think refactorization of the toolchain is no dependency so I welcome everyone to commit in this part (propably as dedicated project), too!
This is probably obvious to everyone in the discussion, but I suspect the Raspberry Pi 4 or 5 will have 2GB of RAM or more. And there are other single board computers only marginally more expensive than the Pi that have 2GB already. Any guesses as to whether that's sufficient for a Sandstorm installation? Or even then would users need to restrict the amount of concurrent applications they use?
I've got sandstorm running on an x86_64 box with 2GiB of ram, so that's good enough for my own use. I'm pretty much the only user though.
Yes, people have reported success with 2GB. Note that the amount of RAM you need is proportional to the number of active users -- or, really, the number of active grains. (We have plans to improve the situation someday using https://github.com/sandstorm-io/snappy-start but not sure when that will happen.)
Quoting Kenton Varda (2017-01-27 02:35:06)
Take a look at the Archive type at the bottom of package.capnp -- that's the actual package format.
Curious, what's the difference between that and the actual .spk file? capnp decode tells me it's not a capnp message, and It's apparent from the file sizes I've seen that there's some compression going on as well.
I think work on such tooling could be a pretty interesting project that someone could easily own and work on without requiring lots of coordination with other parts of the project, if someone is interested!
Might be up for it if I can find some time, but I can't really promise much at the moment. I have a separate interest in reproducible builds as well, so all of the magic in the current tooling is something that is very much the opposite of what I want :/.
Curious, what's the difference between that and the actual .spk file?
See the comments on stuff defined right above "Archive" -- the full file is actually two capnp messages (Signature and Archive), xz-compressed, with an 8-byte "magic number" prefix.
i've meddled with ekam and building on rpi3 arm64 , tends to eat disk.... however looking to get a rockpro64 for home soon , could try to build on cloud.scaleaway. com .. as i have 8 gigs... of ram etc...
I have a khadas vim2 with 3gb of ram, and (if I can find the time) I'm going to try to get sandstorm running on it.
BTW, Wekan is made with Meteor, and Wekan already runs on RasPi3 and RasPi4 on arm64 and any other Node.js/MongoDB supported CPU architecture:
I think the largest problem is that even though a lot of dependencies for Sandstorm probably support ARM now, that we have a hard enough time getting apps packaged and updated for Sandstorm right now. Adding a CPU architecture would require all the apps get packaged for it as well.
I think it's worth putting some thought into what implications multiarch has for the build pipeline, rather than get something working for x86_64 and hope it still makes sense elsewhere. I haven't investigated what multiarch looks like in the docker world, I should find some time to read up on where things are there and what that means for docker-spk's prospects of being able to handle this.
@zenhack
Yes, I really would like to figure out multiarch Wekan Docker and Snap. I have tried but I will sometime try to research more, I don't have it working yet.
I don't know would is be possible to run Sandstorm in Docker or Snap.
ekam will build on arm64 , which is needed for some of the building.. some time ago for captnproto however for rpi3 or less , you may need to make -j2 1 etc lowering memory foot print. {however (https://www.scaleway.com/en/virtual-instances/arm-instances/) debian native , 8-16 gigs of ram should be easy to build and exportbuilt artifacts ... to RPI etc... }
https://github.com/capnproto/ekam "Ekam's primary user and maintainer right now (Sandstorm.io) is itself highly Linux-specific, so there is not much pressure. (Let us know if you want to help.)"
I was thinking about this again recently. For dev tooling, we could likely make use of the kernel's binfmt_misc with qemu user mode to create a vagrant vm that can be used to develop arm packages on an x86_64 host. This way devs could use their own machines (even if they don't have access to an arm box) and much of the vagrant-spk workflow would be the same. This still leaves devs needing to test each arch manually though.
Does something need building on arm64? I have access to bare metal arm64 sever that has 125 GB RAM and CPU that has 96 cores.
@zenhack I think the biggest issue is with already having difficulty getting people to update their packages, asking them to make two packages and update both is pretty painful. My trauma from Windows on ARM-related travesties comes to mind.
I'd want our vagrant-spk scripts to have well-documented methods for splitting out "on arm do this, on x64 do this", so one doesn't have to maintain fully separate scripts for each architecture, despite the fact that a lot of the build instructions are probably architecture-specific. Presumably we'd want vagrant-spk devs to pass an arm argument somewhere in their vagrant-spk commands to trigger testing and packing the ARM version of a given package.
By default, meteor only supports x64 (and maybe x32). For Wekan, I first create x64 bundle .zip file, and then make changes to it with maintainer bash script: https://releases.wekan.team/raspi3/
For arm64, MongoDB 3.x is installed from Ubuntu repos.
For IBM Z Series mainframes, I only add fibers, and use it with Node and MongoDB for s390x: https://github.com/wekan/wekan/wiki/s390x#install