Docker Hub Image

[bytesnz]

Just starting to add remote-storage (RS) to some projects I am working on, so am setting up my own RS server. I didn't see that you don't have a (recent) docker image on docker hub, so created a Dockerfile and a Github Action to build and publish one when a release is created.

The image is very basic - uses the node:lts-alpine docker image and the basic armadietto config (from armadietto -e). It exposes the armadietto http port to be used with a proxy.

Are you interested in including this in the repo? Figure you could create an organisation on docker hub to publish it under.

Need to at least:

• [x] figure out specifying the server url for server over a proxy (need to ensure X-Forwarded-Host is set)
• [x] complete github action file (or complete later)
• [x] agree on Dockerfile local copy vs yarn global / npm i -g
• [x] add tests to github action file

[raucao]

Anyone reading this has enough Docker experience/knowledge to be able to review this PR? Any and all help would be most appreciated! :pray:

[peponi]

I prefer standalone docker files, i.e. git clone within the image build and not before

reduces build steps and errors on the end user side

but this is not needed for this project IMHO because there is already a finished package on npmjs

I would suggest using this

My Dockerfile

FROM alpine:latest AS appBuild

LABEL maintainer="Peponi <[email protected]>" \
      description="will run the Armadietto NodeJS web service (a RemoteStorageJS backend)"

ARG PROJECT_NAME="armadietto"
ARG PORT="8000"
ARG PKG_MANAGER="yarn"
ARG INSTALL_COMMAND="yarn global add"
ARG CONFIG_PATH_CERTS="/etc/letsencrypt/live/example.com/"
ARG CONFIG_PATH_STORAGE="/usr/share/armadietto"

ENV PROJECT_NAME=$PROJECT_NAME

RUN set -e;\
  apk add  \
    curl \
    git \
    nodejs \
    $PKG_MANAGER; \
  mkdir /opt/armadietto; \
  mkdir -m 0700 /$CONFIG_PATH_STORAGE; \
  mkdir -m 0700 /$CONFIG_PATH_CERTS; \
  $INSTALL_COMMAND $PROJECT_NAME; \
  apk del git $PKG_MANAGER; \
  adduser -u 6582 -HD $PROJECT_NAME;

COPY config.json /etc/armadietto.conf.json

VOLUME $CONFIG_PATH_STORAGE $CONFIG_PATH_CERTS
USER $PROJECT_NAME
WORKDIR ~
EXPOSE $PORT

CMD $PROJECT_NAME -c /etc/armadietto.conf.json

HEALTHCHECK CMD curl --fail http://127.0.0.1:$PORT/ || exit 1

### Install ###
#
# BUILD:
#
# default for amd64 architecture
#
# > docker build -t armadietto:latest .
# > docker build -t --build-arg PKG_MANAGER="npm" --build-arg INSTALL_COMMAND="npm i -g" armadietto:latest .
#
# RUN:
#
# > docker run --rm -p 8000:8000 rarmadietto:latest
#
# INFO: config.json needs to be byside the Dockerfile
#
# {
#   "allow_signup": true,
#   "storage_path": "/usr/share/armadietto",
#   "cache_views": true,
#   "http": {
#     "host": "0.0.0.0",
#     "port": 8000
#   },
#   "https": {
#     "enable": false,
#     "force": false,
#     "port": 4443,
#     "cert": "/etc/letsencrypt/live/example.com/cert.pem",
#     "key": "/etc/letsencrypt/live/example.com/privkey.pem"
#   },
#   "basePath": ""
# }
#
###

e2e test

image size

EPOSITORY                                     TAG                 IMAGE ID            CREATED             SIZE
armadietto                                     peponi              9a84b73851eb        21 minutes ago      52.4MB
armadietto                                     bytesnz             9829e4ad797b        About an hour ago   118MB

security scan with trivy

 ✘  ~/Code/private/DockerFiles/armadietto  docker run --rm  -v /var/run/docker.sock:/var/run/docker.sock -v cache:/root/.cache/ aquasec/trivy armadietto:peponi
2021-05-23T09:21:40.833Z        INFO    Detected OS: alpine
2021-05-23T09:21:40.833Z        INFO    Detecting Alpine vulnerabilities...
2021-05-23T09:21:40.836Z        INFO    Number of PL dependency files: 1
2021-05-23T09:21:40.836Z        INFO    Detecting yarn vulnerabilities...

armadietto:peponi (alpine 3.13.5)
=================================
Total: 0 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 0)


usr/local/share/.config/yarn/global/yarn.lock
=============================================
Total: 0 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 0)

[raucao]

@peponi That looks great! :clap:

@bytesnz Want to review @peponi's proposed changes to the Dockerfile?

[raucao]

created a Dockerfile and a Github Action to build and publish one when a release is created

I just created a remotestorage organization on Docker Hub and then created a repo for armadietto. However, I cannot find where to create custom access tokens, and the permissions tab doesn't let me use the "owners" team to create a new permission entry.

Maybe someone more experienced wants access to the RS team there? The free/community plan allows 2 more accounts to be added...

Edit: considering that we're stuck on GitHub for now, would it also make sense to consider GitHub's own Docker registry for publishing this? Seems to me like Docker Hub is rather crippled on free plans. Edit 2: Looks a bit complicated to me.

[bytesnz]

I prefer standalone docker files, i.e. git clone within the image build and not before

reduces build steps and errors on the end user side

but this is not needed for this project IMHO because there is already a finished package on npmjs

I would suggest using this

My Dockerfile

FROM alpine:latest AS appBuild

LABEL maintainer="Peponi <[email protected]>" \
      description="will run the Armadietto NodeJS web service (a RemoteStorageJS backend)"

ARG PROJECT_NAME="armadietto"
ARG PORT="8000"
ARG PKG_MANAGER="yarn"
ARG INSTALL_COMMAND="yarn global add"
ARG CONFIG_PATH_CERTS="/etc/letsencrypt/live/example.com/"
ARG CONFIG_PATH_STORAGE="/usr/share/armadietto"

ENV PROJECT_NAME=$PROJECT_NAME

RUN set -e;\
  apk add  \
    curl \
    git \
    nodejs \
    $PKG_MANAGER; \
  mkdir /opt/armadietto; \
  mkdir -m 0700 /$CONFIG_PATH_STORAGE; \
  mkdir -m 0700 /$CONFIG_PATH_CERTS; \
  $INSTALL_COMMAND $PROJECT_NAME; \
  apk del git $PKG_MANAGER; \
  adduser -u 6582 -HD $PROJECT_NAME;

COPY config.json /etc/armadietto.conf.json

VOLUME $CONFIG_PATH_STORAGE $CONFIG_PATH_CERTS
USER $PROJECT_NAME
WORKDIR ~
EXPOSE $PORT

CMD $PROJECT_NAME -c /etc/armadietto.conf.json

HEALTHCHECK CMD curl --fail http://127.0.0.1:$PORT/ || exit 1

### Install ###
#
# BUILD:
#
# default for amd64 architecture
#
# > docker build -t armadietto:latest .
# > docker build -t --build-arg PKG_MANAGER="npm" --build-arg INSTALL_COMMAND="npm i -g" armadietto:latest .
#
# RUN:
#
# > docker run --rm -p 8000:8000 rarmadietto:latest
#
# INFO: config.json needs to be byside the Dockerfile
#
# {
#   "allow_signup": true,
#   "storage_path": "/usr/share/armadietto",
#   "cache_views": true,
#   "http": {
#     "host": "0.0.0.0",
#     "port": 8000
#   },
#   "https": {
#     "enable": false,
#     "force": false,
#     "port": 4443,
#     "cert": "/etc/letsencrypt/live/example.com/cert.pem",
#     "key": "/etc/letsencrypt/live/example.com/privkey.pem"
#   },
#   "basePath": ""
# }
#
###

e2e test

image size

EPOSITORY                                     TAG                 IMAGE ID            CREATED             SIZE
armadietto                                     peponi              9a84b73851eb        21 minutes ago      52.4MB
armadietto                                     bytesnz             9829e4ad797b        About an hour ago   118MB

security scan with trivy

 ✘  ~/Code/private/DockerFiles/armadietto  docker run --rm  -v /var/run/docker.sock:/var/run/docker.sock -v cache:/root/.cache/ aquasec/trivy armadietto:peponi
2021-05-23T09:21:40.833Z        INFO    Detected OS: alpine
2021-05-23T09:21:40.833Z        INFO    Detecting Alpine vulnerabilities...
2021-05-23T09:21:40.836Z        INFO    Number of PL dependency files: 1
2021-05-23T09:21:40.836Z        INFO    Detecting yarn vulnerabilities...

armadietto:peponi (alpine 3.13.5)
=================================
Total: 0 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 0)


usr/local/share/.config/yarn/global/yarn.lock
=============================================
Total: 0 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 0)

Cool. Looks good. Will integrate some of the stuff with the MR. The reason I went for using copy instead of yarn install is for integrating with github actions I - it would need to be published to npm before the docker image is created, but that's doable.

I think it is good to have a docker image as well as it makes it super easy to deploy.

[AngeloR]

We may want to utilize GitHub actions to do the final build/push to docker hub as well. I'd be in favour of remaining on docker hub, but if we want we can always push the image to multiple locations.

[bytesnz]

We may want to utilize GitHub actions to do the final build/push to docker hub as well. I'd be in favour of remaining on docker hub, but if we want we can always push the image to multiple locations.

:+1: The MR has a template Github Action for building docker images, but it will need some tweaking.

[raucao]

@bytesnz Did you find some time to integrate the things you wanted by any chance? No pressure, just making sure the PR doesn't go stale...

[bytesnz]

@raucao I did start making changes and looking at it. I unfortunately ran out of time before going on a trip. I was going to merge the two dockerfiles and add testing to the action from peponi. I can do this at the start of Sep (long trip without a laptop).

[bytesnz]

Sorry about the delay. Have done a bit of merging between the two Dockerfiles:

• basing it off alpine with a build container, I got it down to 49MB
• went for a local copy based approach thinking it would be good doing things like making images from branches, during dev for eg
• went for wget instead of curl for the healthcheck (currently won't work if https is enabled though I think)
• added trivy to the docker build job

For the github actions job, would be good to add an e2e test run to it as well, but would require a bit of test before rewriting I think (as each spec is starting up a server). Would also be good to split the job up into build, test, security test, but would need to either push a temporary image somewhere (or use some blugh caching I think).

[JakubNer]

@bytesnz, was testing this a bit. Pulled your branch and...

docker build -t remotestorage/armadietto:latest -f .\docker\Dockerfile .
docker run -d -p 8000:8000 remotestorage/armadietto:latest

But it exits right away with ': No such file or directory in the logs.

So I went into the image:

docker run -it --rm -p "8000:8000" remotestorage/armadietto:latest /bin/sh

And running the linked /opt/armadietto/bin/armadietto.js conks out in the same manner.

Are we certain we can start the process like that?

I've only started it as a library as per /example/server.js

Would love to get this out and build on top of it. I see there are about 15k downloads of armadietto on docker-hub from various providers, would be nice to get this official one out 👍.

[bytesnz]

Hey @JakubNer. Hmm, that's weird. Just tried running through the build process again and no issues for me (commands below). Is the image bytesnz/armadietto:latest working for you (is built off this repo?

git clone https://github.com/bytesnz/armadietto
cd armadietto/
git checkout  add-dockerfile 
docker build -t remotestorage/armadietto:latest -f ./docker/Dockerfile .
docker run -p 8000:8000 --name armadietto remotestorage/armadietto:latest

[JakubNer]

@bytesnz, indeed when I pull your image (bytesnz/armadietto:latest) off docker hub, works like a charm 👍.

My own build fails like I mentioned. I did build off of the right place:

PS C:\jj\src\armadietto> git status
On branch add-dockerfile
Your branch is up to date with 'bytesnz/add-dockerfile'.

Just rebuilt it locally, same error.

Weird.

[JakubNer]

@bytesnz, I'm puzzled. I checked your docker-hub (https://hub.docker.com/r/bytesnz/armadietto/tags) and the image that runs for me is the image the git actions built and pushed last Octo (https://github.com/bytesnz/armadietto/actions/runs/1385063021) which is the same commit as of this PR and your tip-of-tree in your branch.

So I don't understand why the linked /usr/local/bin/armadietto invocation run inside that image doesn't execute when I re-build locally.

What always works is simply removing the link on line 44 of the Dockerfile and replacing line 52 with:

CMD node /opt/armadietto/bin/armadietto.js -c /etc/armadietto.conf.json

I wonder if the difference is my host. I'm building on Windows in PowerShell. I don't see how that should make a difference though.

[raucao]

I thought it might help if someone else tested this to confirm, so I tried the instructions from https://github.com/remotestorage/armadietto/pull/37#issuecomment-1004570628 on Arch Linux, and it worked without issues.

However, if it works on Windows without the symlink (which I also have no idea how to explain), I guess forgoing it in favor of using the full path in the final command wouldn't hurt anyone.

[bytesnz]

So I don't understand why the linked /usr/local/bin/armadietto invocation run inside that image doesn't execute when I re-build locally.

@JakubNer oooo that is weird. If you shell into your created docker image does the link exist and does it point to where it should?

/opt/armadietto $ ls -l /usr/local/bin/
total 0
lrwxrwxrwx    1 root     root            33 Oct 26 09:51 armadietto -> /opt/armadietto/bin/armadietto.js

I'm also curious to know what happens when you try and create a symbolic link in it

/opt/armadietto $ ln -s /opt/armadietto/bin/armadietto.js /tmp/
/opt/armadietto $ ls -l /tmp/
total 0
lrwxrwxrwx    1 armadiet armadiet        33 Jan 16 20:52 armadietto.js -> /opt/armadietto/bin/armadietto.js

But yes agreed, if the full path works, we should just switch it to that.

[JakubNer]

@bytesnz

If you shell into your created docker image does the link exist and does it point to where it should?

Indeed it exists but produces the same ': No such file or directory output:

PS C:\jj\src\overhide-ledger> docker run -it --rm -p "8000:8000" remotestorage/armadietto:latest /bin/sh
/opt/armadietto $ ls -l /usr/local/bin/
total 0
lrwxrwxrwx    1 root     root            33 Jan  2 07:59 armadietto -> /opt/armadietto/bin/armadietto.js
/opt/armadietto $ armadietto
': No such file or directory

Relinking works and lists in /tmp.

[raucao]

@bytesnz What's the status of the open tasks here? I see some unchecked ones about GitHub Actions, which I assume is about automatically building new images upon merging or for releases, right?

[bytesnz]

In terms of the tasks:

• complete github action file (or complete later)

  Looking at now (https://github.com/bytesnz/armadietto/compare/add-dockerfile...bytesnz:expand-docker). Looking to:

  • run the tests in the action
  • allow the workflow to run on all pushes (except for the publish)

  The only issue is nicely running the e2e tests. Looking at the specs, they create in instance of armadietto themselves rather than querying an external server instance. Not sure if there is a way to run the tests on an external server, but was thinking could either:

  • mount and run the tests in the created docker container, or
  • modify the tests so that there is an option to run the tests as they are or against an external server

• add tests to github action file

  See above

• agree on Dockerfile local copy vs yarn global / npm i -g

  Silently looking for agreement/protest - currently using local copy for the benefit of being able to run the actions on non-published versions.

[raucao]

Cool, thanks.

Not sure I understand the problem with the tests. There's already a working Action for that. Shouldn't they each do their own thing and simply kick off other Actions depending on success? I don't really know much about GitHub Actions, but pipelines composed of different workflows are pretty standard in CI these days.

Edit: what I mean by that is that the process should be something like: "test & lint" -> "publish npm package" -> "build docker image" -> "publish docker image"

As we don't have a good process for automatic versioning, these steps would probably not apply for every merge per se. The npm version would be manual, and pushing a new version, or publishing release notes for it, would probably trigger the npm publish and Docker publishing.

[bytesnz]

With the tests, I was thinking it would be good idea to test the docker image to ensure the image is working as expected (the docker configuration is correct and the ports/volumes are working as expected). I was thinking about using the e2e tests just as they are already written, but the test could just be a request to the running docker image to ensure it returns the expected response.

For the release, the current action will run when a release is created. The one I was working on was running on any push, but it only publishes the image when a tag is created. Can align it to the current tests (run on master pushes and release creation).

[raucao]

image to ensure the image is working as expected (the docker configuration is correct and the ports/volumes are working as expected).

Ah, I see now. I don't think we have to run the normal test suite for that.

For the release, the current action will run when a release is created. The one I was working on was running on any push, but it only publishes the image when a tag is created. Can align it to the current tests (run on master pushes and release creation).

That sounds reasonable to me. If we build images for all tags, then we can test beta tags directly from the registry before tagging stable releases.

[bytesnz]

That sounds reasonable to me. If we build images for all tags, then we can test beta tags directly from the registry before tagging stable releases.

Yeah. Ah... just noticed there are no tags or releases at the moment. Is this something that is being looked at?

[raucao]

Yes, definitely. We discussed Armadietto in the monthly hangout call today and had rough consensus among the people on the call about tagging a new release soon.

@lesion I think you are the one in control of publishing to npm, right? Wondering if you want to help with the release or how we should go about it.

[JakubNer]

@bytesnz,

agree on Dockerfile local copy vs yarn global / npm i -g

What does this mean? Is this some alternative to what you did whereby we'd use's npm's registry or some such? I don't see why we'd not want to do what you did already.

[JakubNer]

@bytesnz, made a PR on this branch for your consideration. Maybe a way forward with running the tests against the docker image. All the tests pretty much already use chai-http so the change was small.

Unfortunately the oauth tests have some storage issue, couldn't figure it out yet. Webfinger and file tree tests pass OK.

The PR also has a change to replace the symbolic link with a direct node CMD so I can run builds built on my machine (Windows).

To run, I do (powershell):

// first console
docker build -t remotestorage/armadietto:latest -f ./docker/Dockerfile .
docker run --rm -p 8000:8000 --name armadietto remotestorage/armadietto:latest

// second console
${env:TEST_PORT}=8000
npm run test

All from repo root. So I think running tests like this should be doable from the github action workflow by just setting the TEST_PORT env variable. But, need to figure out the test failures. I didn't get any issues when creating account from the /example/index.html against this image.

[bytesnz]

What does this mean? Is this some alternative to what you did whereby we'd use's npm's registry or some such? I don't see why we'd not want to do what you did already.

@JakubNer, yeah, @peponi suggested using the published package for the dockerfile (quote below). I think the doing it from git will give more options in the future (eg creating docker images from non-published versions

but this is not needed for this project IMHO because there is already a finished package on npmjs

@bytesnz, made a PR on this branch for your consideration. Maybe a way forward with running the tests against the docker image. All the tests pretty much already use chai-http so the change was small.

Cool, yeah that looks good. Sorry I hadn't update the command already (got distracted with the reworking the action to add the testing). Have added the CMD change to the temp branch I'm working, which I will hopefully finish in the next few mins.

For the testing, yeah that looks good - was thinking of something similar, though was thinking we could extract the start/stop logic into its own helper file, so it only needs to be changed once. Weird about the tests not passing. Can try and have a look at some point

[bytesnz]

Ok. So, I have rewritten the github action so that it is split into multiple jobs. This cleans it up a bit and allows tests to be run in parallel (multiple jobs).

The jobs are:

• version - detects the version that should be used for tagging the docker image (GITHUB_REF which will be a tag when it is running on a release
• build - build the docker image and create an artifact from the build
• sec_test - runs the docker image against aquasec/trivy
• e2e_test - will run the tests to make sure the image is working as expected (currently commented out)
• publish - publishes the docker image (only runs on a tag [when a release is published])

Things of note:

• The sec_test job won't fail if it detects bad stuff... not sure if we want to enable it. Would be good to if it fails, mark it as a warning... but I am newish to actions, so not sure how to do this yet
• The action needs to be set up as a check to be helpful for PRs... again not sure how to do this yet (what can I say, I'm a gitlab user).

As it is a big change, haven't added it to this branch yet, it is currently in its own branch, but I can merge it if people are happy with the new format https://github.com/bytesnz/armadietto/compare/add-dockerfile...bytesnz:expand-docker?expand=1

[JakubNer]

I'd vote for you to merge it. The pipeline runs well and produces an image that I just ran through the /example/index.html adhoc tests on local and does OK.

I wouldn't worry too much that the e2e job is commented out. We can iterate.

[lesion]

Yes, definitely. We discussed Armadietto in the monthly hangout call today and had rough consensus among the people on the call about tagging a new release soon.

@lesion I think you are the one in control of publishing to npm, right? Wondering if you want to help with the release or how we should go about it.

As you may have noticed, I don't have time to track your wonderful progress in remotestorage world, so I'll be happy to transfer the package to someone else -> https://docs.npmjs.com/transferring-a-package-from-a-user-account-to-another-user-account