What about docker ?

[justUmen]

I appreciate the idea and good luck with that ! But with a docker image, it will be so much better. ;) You probably have your reasons to stick with nix, but it makes this project quite difficult to get into.

[aszlig]

@justUmen: Yeah, at some point I do plan to add a Docker image. However the problem at the moment is that the source code is patched accordingly depending on the NixOS configuration. One example for this is that https://habitica.com/ is hardcoded in a lot of places (see source.nix).

So while nixpkgs has tooling for building Docker images, you'd then still need Nix to build the Docker image if you don't want to end up with eg. a BASE_URL of http://localhost:3000, which is not very useful for production environments.

[justUmen]

I see... I tried nix-shell build-vm.nix on 2 machines, both time error: Nix daemon out of memory, how much memory is needed for this ?

[aszlig]

@justUmen: Try nix-build build-vm.nix. You should get a symlink called result in the current directory which you can use to start the VM with ./result. Note that the build-vm.nix currently is only suitable as a development/testing VM.

[justUmen]

Thank for the asnwer. Failed again on the 2Gb machine, out of memory. On a 4Gb machine it is working but crash eventually after running the compilation for a long time.

Install file: "build/nixos/mongo/mongos" as "/nix/store/bshnkp68crs29gagvlrl93anz6166g3f-mongodb-3.4.10/bin/mongos"
chmod 755 /nix/store/bshnkp68crs29gagvlrl93anz6166g3f-mongodb-3.4.10/bin/mongos
collect2: error: ld returned 1 exit status
scons: *** [build/nixos/mongo/mongoperf] Error 1
scons: building terminated because of errors.
build/nixos/mongo/mongoperf failed: Error 1
builder for '/nix/store/0zx3ixy3hnsd8q110rrxl0pah5s0xhmc-mongodb-3.4.10.drv' failed with exit code 2
building '/nix/store/008jrf82v2dmcz6bczgh1z9y56rll12w-mongodb-core-2.1.19.tgz.drv'...
cannot build derivation '/nix/store/ffjh2q777dh9f0akg96y6ghqbizcs1ln-unit-shabitica-db.service.drv': 1 dependencies couldn't be built
building '/nix/store/fsy2rqi5jw49h9dxvzsb07fflgpiwhr5-mongodb-core-3.0.8.tgz.drv'...
cannot build derivation '/nix/store/d6ad3wsxfvjy2llckxcmfx9nxviz9rx7-system-units.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/kl1yvw4r2ghysmbgmgq716x9g8p00r4w-etc.drv': 1 dependencies couldn't be built
building '/nix/store/55x993b90anlkd69mlxvii762cddsmyv-mutt-1.10.0.drv'...
cannot build derivation '/nix/store/9n7yzvys51668vxm5j7m9q20k8a88skq-nixos-system-shabitica-dev-18.09pre143632.d91c70171fb.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/01qsd7kigjlnsrp0s70587gqhwb19dmr-nixos-vm.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/5wfhjxmq7473bissxgxgp89bgqy3mv5w-run-vm.drv': 1 dependencies couldn't be built
error: build of '/nix/store/5wfhjxmq7473bissxgxgp89bgqy3mv5w-run-vm.drv' failed

I guess I'll be waiting patiently for the docker version. :)

[aszlig]

@justUmen: Ah, right, this is because Shabitica uses a patched MongoDB, so there are no binaries available and thus it's compiling from source. I probably should mention the binary cache (https://headcounter.org/hydra/) in the README. Compiling MongoDB takes a while and the linking process is quite memory hungry.

So if you want to fetch the binaries instead, you could add headcounter.org:/7YANMvnQnyvcVB6rgFTdb8p5LG1OTXaO+21CaOSBzg= to nix.conf, like this:

trusted-public-keys = headcounter.org:/7YANMvnQnyvcVB6rgFTdb8p5LG1OTXaO+21CaOSBzg= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=

After that you can build with nix-build --option substituters https:/headcounter.org/hydra/ build-vm.nix and it should fetch the binaries instead of compiling from source. Note however, that fetching binaries from random places (yes, this includes Docker images) is something I wouldn't recommend because source code is better to audit, but I guess that's just my personal preference.

Apart from that... even with Docker, you'll still need Nix to build the Docker image because of the aforementioned reasons.

I guess I'll be waiting patiently for the docker version. :)

Waiting is probably a good idea anyway, because as mentioned: build-vm.nix is not for production and also contains a lot of stuff that you don't need for Shabitica (like eg. my patched Vim and a patched Mutt along with a custom Postfix configuration), so evaluation itself can already get quite memory hungry (not as much as linking MongoDB though).

[WhiteNoseThunderPants]

Just wanted to throw my hat in the ring for docker support. I'm going to attempt to get everything running as it currently is, but docker would make things much easier.

[nathan-sain]

@aszlig have you had any more time to consider a docker container for this? I'm not sure I can afford another server just to run Nix on and I would like to run this for my family.

[aszlig]

@nathan-sain: Unfortunately not yet, but you can also deploy build a NixOS VM with Nix (the package manager), like mentioned in a previous comment. If there would be an easy way to build a systemd-nspawn container or a VM with a production setup, would that also work for you?

[nathan-sain]

I'm currently running several services in docker containers on an Ubuntu 18.04 server with 4GB of RAM. This version of Ubuntu includes support for systemd-nspawn so I think the that would be the best way to proceed.

[jtagcat]

Any updates?

[aszlig]

@jtagcat: Not yet, since at the moment I haven't even rebased against upstream since they made some fundamental changes that require more work to properly integrate (probably a full weekend or more), which I currently don't have :-/

[jtagcat]

@jtagcat: Not yet, since at the moment I haven't even rebased against upstream since they made some fundamental changes that require more work to properly integrate (probably a full weekend or more), which I currently don't have :-/

Me neither, extremely low on time. I would pay for their hosting, but I already felt that they had way too many privacy issues.

Since I don't know about habitica's code and rn don't want to have anything new, would you accept a bounty?

[aszlig]

Since I don't know about habitica's code and rn don't want to have anything new, would you accept a bounty?

Well, same here, since I don't know what has changed since the last update.

I'm also not sure whether a bounty would help a lot here. The rebase for the current upstream version is something that's still in my todo list, so eventually it will happen at some point anyway and a small bounty wouldn't make a difference in schedule.

However, when it comes to supporting Docker, it's an entirely different thing. Just doing the initial work will not be enough because I'll be dogfooding only the NixOS service and would probably constantly break Docker.

So for that I guess someone who's knows Nix, is using Docker and wants to maintain it long-term would be a better canidate. Until that, it's going to be on my todo list after the upstream update mentioned above.

[Don-Swanson]

Hello, I'm interested in helping get a docker image going. Can you give me a bit more detail about the locations of anything that would need to be persistent? IE Config files/databases/etc.

[aszlig]

@TheEagle13: Essentially the database is the only thing that needs to be persistent, everything else is entirely disposable.

There are some state files around migrations and the Armoire and secrets, but all those are disposable as well, because deleting them will just result in minor inconveniences (eg. users being logged out).

From the long-running services standpoint, we have the main service exposing the API and it's proxied by nginx (of course only if useNginx is enabled, which it is by default), the client is served statically.

The only communication that is allowed to happen from the main service is to MongoDB and to the mailer daemon (source code), the latter is essentially mimicking ther transactional mail API and glues together the local MTA with the main service. It also provides the email templates that are not included in the upstream source tree.

Another service is the image proxy, which makes sure that images in markdown can't be used to track user behavour and alse to avoid mixed-content warnings. One noteworthy thing here is that the image proxy is patched to decode the Shabitica session to avoid another attack vector that'd make it possible to brute-force the MAC. The image proxy is also reverse-proxied via nginx and is sandboxed as all the other services and only reachable via Unix domain socket.

The last essential service is the database update service, which is there to make sure that whenever there is a code update, the database is also migrated accordingly (after doing a backup of the database, as a fallback if things go south), because upstream is running migrations manually and you also have to follow development to figure out which migrations are needed.

All of these services (except the latter, which is one-shot) are communicating solely via Unix domain sockets and use their own networking namespace so that the attack vector is pretty much only nginx, even for non-privileged local users.

One point to note here is that the main service and the client are built from a common source package, which is configured via the NixOS options, so for example BASE_URL and ADMIN_EMAIL are patched in after our main patches are applied.

This is of course only a very rough overview and there is a lot more tooling involved, especially when it comes to testing, where we have dedicated E2E tests, migration tests and a few VM-based tests but also running the upstream tests. Apart from that, we also have miscellaneous tooling for maintenence, eg. database access and backup/restore utilities along with their services.

[jtagcat]

Thoughts on throwing this or at least link of this issue to a .md (contributing, readme, wherever a contributor might see it)?