RFC: Customizing the default server Dockerfile

[shayneczyzewski]

Overview

Currently, when users run wasp build, Wasp will write a Dockerfile to:

.wasp/build/
├── Dockerfile 👈
├── db
├── installedFullStackNpmDependencies.json
├── server
└── web-app

The contents of this file are based on a template and can change based on what features they are using (e.g. Prisma, or Auth with NPM deps requiring patches).

The problem is users cannot modify this file during generation, they can only change it after the fact. However, upon subsequent builds, these changes will get overwritten.

What we would like to provide is a way for users to specify changes to our Dockerfile once in their .wasp file, and have that take effect during generation.

Initial idea

Initially, I was thinking this would be easy, we can just have the users provide a path to some Dockerfile relative to @ext, and if present, we will use their file instead of ours. However, as noted, the contents of our file may change as their feature usage changes, making it tricky for them to stay up to date. Also, if we changed the source template, they would have to know to review those changes for incorporation. Therefore, I nixed this.

Current idea

A bit of background is required before I go into the current idea.

Background

Our current Dockerfile makes heavy use of stages. We have one for the base, one for building our server code, and one that we actually run and makes use of the build artifacts.

For reference, here is what our Dockerfile output looks like for waspc/examples/todoApp:

FROM node:16-alpine AS node

FROM node AS base
RUN apk --no-cache -U upgrade # To ensure any potential security patches are applied.

FROM base AS server-builder
# Install packages needed to build native npm packages.
RUN apk add --no-cache build-base libtool autoconf automake python3
WORKDIR /app
# Install npm packages, resulting in node_modules/.
COPY server/package*.json ./server/
RUN cd server && npm install
COPY db/schema.prisma ./db/
RUN cd server && npx prisma generate --schema=../db/schema.prisma

# TODO: Use pm2?
# TODO: Use non-root user (node).
FROM base AS server-production
ENV NODE_ENV production
WORKDIR /app
COPY --from=server-builder /app/server/node_modules ./server/node_modules
COPY server/ ./server/
COPY db/ ./db/
EXPOSE ${PORT}
WORKDIR /app/server
ENTRYPOINT ["npm", "run", "start-production"]

One neat thing you can do is override certain stages, or the ENTRYPOINT itself. The last (bottommost) declaration wins (TODO: I need to verify this is defined in the spec and not just a happy coincidence). Here is a file you can play around with to verify this:

# To test, save this file and run:
# $ docker build -t dockerfiletest . && docker run --rm dockerfiletest
# Uncomment the options to see how it changes.

FROM alpine as base
ENV BASE true
RUN echo "base"

FROM base as build
ENV BUILD true
RUN echo "build"

FROM build as exec
ENV EXEC true
RUN echo "exec"
ENTRYPOINT ["env"]

##### We would append the user Dockerfile contents below ours to allow for ergonomic overrides.

# (Option 1) You can change the ENTRYPOINT like so:
# FROM exec
# ENV CUSTOM true
# RUN echo "option 1"
# ENTRYPOINT ["ls"]

# (Option 2) You can overwrite a stage (e.g. exec) like so:
# FROM build as exec
# ENV CUSTOM true
# RUN echo "option 2"
# ENTRYPOINT ["env"]

Back to the new idea

Given this, what if we just concatenated their Dockerfile to the bottom of ours? In that way, they could selectively change various stages, or the ENTRYPOINT. It would be nice to help them see what the Wasp generated Dockerfile would be, so I propose we give them a new CLI command like: wasp dockerfile show that will show what our Dockerfile would be (not including their contents).

This way, if they override a stage, they could ensure they did all the "normal" stuff we would do.

Options

So how can we get the contents of their Dockerfile below ours during Generation?

a) Inject into the template by adding contents to AppSpec

Since we have just a String file path, we need to get the contents somehow if we wanted to inject it into the template. We cannot do this in the normal genDockerfile :: AppSpec -> Generator FileDraft place since we are not in IO here (which is good!). Instead, we'd need to read it from somewhere earlier, where we do have access to IO. One place would be during AppSpec creation.

b) Append one file to another by creating a new FileDraft

Since we are effectively just appending one file to another, that feels like it could be useful in other circumstances in the future too. What if we created a new FileDraft type called AppendFileDraft that took a base File' and a File' to append, and made sure to append the contents during write?

Or maybe something else altogether? Please let me know your thoughts!

[shayneczyzewski]

FYI @Martinsos @sodic as time allows would love your thoughts on the options here. Thanks!

[Martinsos]

@shayneczyzewski nice analysis!

Ok, so the general idea makes sense to me -> we would basically let them add stuff at the bottom of our dockerfile. Let me quickly go through different ideas I can think of:

1. Concatenate commands to the Dockerfile we generate -> you presented this one above.
2. Let them provide their own Dockerfile, as a replacement for our Dockerfile.
3. Provide some kind of "smarter" way to directly change parts of the Dockerfile. So for example they can list additional dependencies, which we will add to the Dockerfile for them then. Or maybe they add additional step which we insert into dockerfile for them.

You suggested (1) as the solution, and also mentioned (2) but decided (1) is better for now.

What feels tricky to me with Dockerfile is that on one hand there is quite some custom stuff in the Dockerfile that we want to set up so that Wasp generated code works, and ideally we don't want them to know about that setup / dependencies, but on the other hand, we also need to allow them quite some flexibility to be able to also do their own custom setup if needed.

It feels pretty hard to satisfy both -> have a way of our own to do some "opaque" setup while also letting them having the flexibility to customize it.

In both (1) and (2), they are somewhat relying on our Dockerfile -> they have to know to some level what we are doing there. So if we make changes to it, there could be an issue. Altough, (1) does feel much more resistant to it in practice, since (2) requires rewriting it, while (1) merely adds upon it.

(3) could be made more robust, but it is also more limiting. Also complicated things a bit maybe by introducing new part of DSL? Hm.

Ok, I am done with my thinking out aloud -> in any case, it seems there are different interesting options, and I don't think it is super clear which one is the winner, both have some pros/cons. With limited information we have, (1) also sounds the best to me as a start. Maybe in the future we could have both (1) and/or (2) as options, or we switch to (3), but I also expected we will learn more in the future and have better ideas what is needed / what people need, so will be able to smarter choose the tradeoffs.

So, the direction regarding Dockerfile sounds good to me!

Regarding implementing this in the Generator -> ha file drafts :D. Yup, we could read it upfront, kind of like we do with ext/, or some other files out there. Or we can add special file draft as you mentioned. That is somewhat specific file draft, so I am a bit tempted to instead push toward reading the file in advance and making it part of AppSpec. If you will do so, pls check how we do the same thing for some other files. Btw this gives you a bit more info as you can parse and do smth with the user supplied dockerfile e.g. do some checks or processing. On the other hand, probably that won't be needed. At the end I don't really know what is better, I would probably explore both options a bit, then switch 5 times between them and end up with a random one hehe so go for whatever you think is best and let's see how it turns out! Ping me if you have any additional questions regarding this.

[shayneczyzewski]

Quick update: I am going down the path of getting access to the Dockerfile during AppSpec construction. The good news is we already have it! :D If we require the Dockerfile to be under the @ext path, which makes sense, then it will automatically be available in AppSpec as one of the externalCodeFiles :: [ExternalCode.File] (which we can later lazily load the contents of). 🥳

So, we just need a way to know which external code file the Dockerfile they want to append is. We could just look for that name, but that feels kinda janky (but still an option). So, we have a few other, more explicit options:

1. dockerfile :: Maybe String - simple, but we don't ensure it is an @ext path at the type level, so we would have to do extra validation checks here. Not a fan of this option.
2. dockerfile :: Maybe ExtImport which looks like dockerfile: import all from "@ext/Dockerfile" - the pro here is we now have a path :: ExtImportPath, but we also confusingly have an name :: ExtImportName and the import syntax doesn't really make sense here for Dockerfiles (but this is how my POC currently works).
3. dockerfile :: Maybe ExtImportPath - this is the direction I am interested in exploring and getting feedback on. I would add new syntax to the DSL so we could do something like dockerfile: file "@ext/Dockerfile" This would give us a new ability to specify a file <ExtImportPath> and I think it may be useful in other places beyond this use case.

Thoughts on the approaches @Martinsos @sodic ? Thanks!

[Martinsos]

@shayneczyzewski all sounds reasonable, nothing to add!

Only other option would be putting Dockerfile somewhere else instead of ext/, for example in the root of the project. We could say if there is Dockerfile there, then we use it. Ok thinking about this now I just also realized that what might be a bit tricky is if you put Dockerfile in ext/, we will copy it both to client and server as we copy all the other files from ext/, right? And that is not really what we want to do. So you would have to add some special rules for Dockerfile when present in ext/ and marked in wasp file as dockerfile? Maybe better to just have special place for it and that is it, like we have for .env files, and it doesn't even have to be in ext/? I guess why ext/ feels a bit smelly is because we are going to be treating Dockerfile in special way. But maybe that is fine hm.

[shayneczyzewski]

Ah, yes, good point @Martinsos. If we had it under @ext it would be copied to both places. It wouldn't hurt anything, but wouldn't be what users expect either, so probably best to keep it where we do for .env files and leave Dockerfile at root to avoid special file processing rules. Then, we wouldn't even need to add it to their .wasp file either. Ok, I'll look into that route instead, thanks!

[Martinsos]

@shayneczyzewski I am good with both directions, but I agree maybe this one is a bit simpler for the start! And we can always make it more flexible if needed.