Running edgeql-js on CI/CD without migration files

[fab1an]

My code looks as follows:

• One repository contains my database and my migrations, it gets deployed using CI/CD.
• Another repository contains a frontend codebase which uses edgeql-js, which also gets deployed using CI/CD.

Locally this is fine, but on CI/CD I cannot build the frontend, since I need to generate the edgeql-js code and I don't have a connection to the database, nor do I have access to the migration-files.

What is the proposed way here?

• Never have your database in a different repository?
• Build edgeql-js in the database-deployment job and publish it somehow?

Thank you for your great work!

[colinhacks]

The easiest thing to do in this situation is check the query builder code into Git so you don't need to worry about regenerating it. We try to generally discourage it but it's the best solution in a situation like this.

I don't have a connection to the database

This is the part that confuses me. Is there a technical reason why you can't connect to the database from your CI here? IF you can connect to the live database there's no need for the migration files. The query builder directly introspects your schema. And since the codebase that relies on the query builder necessarily needs a way to connect to the database in order to execute those queries. Is there a technical reason why isn't it possible to connect to the database in the frontend CI?

[fab1an]

Thanks for the quick answer. I cannot connect to the DB since CI/CD is in the cloud, but database is not reachable from the internet.

[Sikarii]

Another option could be to spawn an ephemeral instance of EdgeDB, apply the migrations and generate the query builder. If you're in the Discord server, check this thread: https://discord.com/channels/841451783728529451/955887885557579788

Below are the 2 Dockerfiles from the thread:

Example 1

FROM node:16-slim AS base

RUN mkdir -p /home/node/app \
&& chown -R node:node /home/node/app

USER node
WORKDIR /home/node/app

COPY package*.json ./
COPY tsconfig*.json ./

# Install node modules
FROM base AS modules
RUN npm ci

# Build EdgeQL-JS by spinning up a live server
FROM edgedb/edgedb:1 AS edgeql-js

RUN mkdir -p /home/edgedb/build/edgeql-js \
&& chown -R edgedb:edgedb /home/edgedb \
&& apt-get update \
&& apt-get install -y curl \
&& curl -sL https://deb.nodesource.com/setup_16.x | bash - \
&& apt-get install -y nodejs

USER edgedb
WORKDIR /home/edgedb/build

COPY --from=base /home/node/app ./
COPY --from=modules /home/node/app/node_modules ./node_modules

COPY dbschema ./dbschema

RUN edgedb-server \
-D /home/edgedb/data \
--security insecure_dev_mode \
--runstate-dir /home/edgedb/runstate & \
  edgedb migration apply \
    -H localhost \
    --tls-security insecure \
    --wait-until-available 2m \
  && npx edgeql-js \
    -H localhost \
    --target cjs \
    --force-overwrite \
    --tls-security insecure \
    --output-dir /home/edgedb/build/edgeql-js

# Build the app
FROM base AS build

COPY --from=modules /home/node/app/node_modules ./node_modules
COPY --from=edgeql-js /home/edgedb/build/edgeql-js ./dbschema/edgeql-js

COPY src ./src

RUN npm run build

# Bring it all together
FROM base AS app

COPY --from=build /home/node/app/dist ./dist
COPY --from=modules /home/node/app/node_modules ./node_modules
COPY --from=edgeql-js /home/edgedb/build/edgeql-js ./dbschema/edgeql-js

CMD ["node", "dist/main.js"]

Example 2

# node and edgedb-js dependencies
FROM node:17-slim AS node-deps
WORKDIR /build
COPY package.json .
RUN echo '{"dependencies": {' $(grep -E -o '"edgedb":\s"(.*)"' package.json) '}}' > package.json && yarn

# ephemeral edgedb and generation
FROM edgedb/edgedb:1 AS edgedb
COPY --from=node-deps /usr/local/bin/node /usr/local/bin/node
RUN mkdir /build && chown edgedb:edgedb /build
USER edgedb
WORKDIR /build
COPY --from=node-deps /build .
COPY dbschema dbschema
RUN edgedb-server \
  -D /build/data \
  --security insecure_dev_mode \
  --runstate-dir /build/runstate \
  & edgedb migration apply \
  -H localhost \
  --tls-security insecure \
  --wait-until-available 2m \
  && node ./node_modules/.bin/edgeql-js \
  -H localhost \
  --target cjs \
  --tls-security insecure \
  --output-dir generated

# build app with generated
FROM node:17-alpine AS app-builder
WORKDIR /build
COPY --from=edgedb /build/generated generated

# ... build your app here

[tdolsen]

There were some changes to this with the v1.0 release, so here's an example updated to use @edgedb/generate, and set to output queries alongside the client as a single file generated/queries.js.

Example v1.0

# edgedb-js dependencies
FROM node:18-slim AS edgedb-dependencies
WORKDIR /build
COPY package.json .
RUN echo '{' \
    '"dependencies": {' $(grep -E -o '"edgedb":\s"(.*)"' package.json) '},' \
    '"devDependencies": {' $(grep -E -o '"@edgedb/generate":\s"(.*)"' package.json) '}' \
    '}' > package.json && yarn
# by grep'ing only edgedb related version we create a caching layer that will only change when edgedb version changes

# ephemeral server and generation
FROM edgedb/edgedb:2 AS edgedb-generate
COPY --from=edgedb-dependencies /usr/local/bin/node /usr/local/bin/node
RUN mkdir /build && chown edgedb:edgedb /build
USER edgedb
WORKDIR /build
COPY --from=edgedb-dependencies /build .
COPY dbschema dbschema
# query files folder - remove or update to your needs
COPY queries queries
RUN edgedb-server \
    -D /build/data \
    --security insecure_dev_mode \
    --runstate-dir /build/runstate \
    > /dev/null 2>&1 \
    & edgedb migration apply \
    -H localhost \
    --tls-security insecure \
    --wait-until-available 2m \
    && node ./node_modules/@edgedb/generate/dist/cli.js edgeql-js \
    -H localhost \
    --tls-security insecure \
    --target cjs \
    --output-dir generated \
    & node ./node_modules/@edgedb/generate/dist/cli.js queries \
    -H localhost \
    --tls-security insecure \
    --target cjs \
    --file generated/queries

FROM node as output
WORKDIR /app
COPY --from=edgedb-generate /build/generated ./generated/edgeql

[scotttrinh]

Going to close this specific issue in favor of discussion within #340 . Feel free to subscribe and/or comment there.