Add environment inference in backend

[rajdip-b]

Description

Accessing environmental variables from using process.env.NAME is a pain. To ease reusability of these environments, and proper inference, we would like to add an object that will fetch the environment values on our behalf.

Solution

The solution looks something like this:

export const Environment = {
  DATABASE_URL: process.env.DATABASE_URL!
}

This is the environment.ts file that will go in apps/api/src/common. The DATABASE_URL is already mentioned in the .env file, all we are doing here is making a proper inference of it.

This issue requires you to do the following things:

• Create a environment.ts in apps/api/src/common
• Add all the environment variable names mentioned in .env.example in environment.ts
• Refactor all usage of process.env.NAME to Environment.NAME

[jamesfrye420]

Why do we need the non-null assertion (!) ?

export const Environment = {
  DATABASE_URL: process.env.DATABASE_URL
}

This works fine as well

[rajdip-b]

Why do we need the non-null assertion (!) ?

export const Environment = {
  DATABASE_URL: process.env.DATABASE_URL
}

This works fine as well

That will matter when we are using the values. You see, those ! marks are to tell typescript that these values will be definitely present.

export const Environment = {
  DATABASE_URL: process.env.DATABASE_URL
}

When we use this, the typescript compiler will complain if we use Environment.DATABASE_URL in some field that requires the DATABASE_URL to be non-null. Hence, we use the !

In cases for our OAuth keys, we can definitely go with GOOGLE_CLIENT_ID: process.env.GOOGLE_CLIENT_ID since the code is configured to not load google oauth if any of the required keys are present.

Hope that clarifies the doubt a bit!

[jamesfrye420]

As of now the process.env.DATABASE_URL resolves to string even if ! is not added. So the ts complier doesn't seem to scream when consuming it.

[rajdip-b]

hmmmmm, thats a consideraton. Well, maybe we can still add them so that developers know what's mandatory and whats not?

[jamesfrye420]

The problem with adding ! is if the process.env.VARIABLE resolves to string | undefined, it will force the engineer to implement fail safe mechanisms for the undefined case. Adding ! would just defy the point of type safety.

The reason for resolving to string instead of string | undefined is cause of the @nestjs/config package (just a guess; not sure).

Regardless of this, mandatory variables imply that the code won't run without them and are absolutely necessary. If that is the case, it makes sense to validate mandatory variables at run time & build time.

As of now the codebase doesn't have any variable which breaks the code if missing so my guess if none of the variables right now is mandatory.

[rajdip-b]

Hmm then maybe we should move to zod validation or the one you shared with me previously?

[jamesfrye420]

There are various ways to do zod validation for env variables. I will list out all some possible ones here:

1. The most obvious is to use the @nestjs/config module.
2. We could use envalid. Though this comes with some limitations since the datatypes are inbuilt with the package and not very flexible.
3. We could use a modify this open source project and tweak it to our needs.

[rajdip-b]

I think we can be NestJS native, and use @nestjs/config

[jamesfrye420]

I propose that we should use the validation to guarantee the vars are present and cast them. Then expose them with a service that is typed.

If the config module validation passes, then we are guaranteed those values that are checked in that validation are present. So we can create a service that consumes the ConfigService that exposes getters that are typed. Or can expose config values that are dynamic and based on other values. Basically, not doing config with process.env at runtime. Validate process.env and read the values into a service that can be used across the application.

Below is an example of the same.

import {Injectable, Logger, OnModuleInit} from "@nestjs/common";
import {ConfigService} from "@nestjs/config";
import {EnvSchema} from "./validate";


@Injectable()
export class ApiConfigService {
    logger: Logger = new Logger(ApiConfigService.name);

    constructor(private config: ConfigService) {
    }

    get env() {
        return this.config.get<EnvSchema["NODE_ENV"]>("NODE_ENV", "production");
    }

    get debug(): boolean {
        const env = this.config.get<EnvSchema["DEBUG"]>("DEBUG", "false");
        return env === "true";
    }

    get databaseUrl(): string {
        return this.config.getOrThrow("DATABASE_URL");
    }

    get secret(): string {
        return this.config.getOrThrow("SECRET");
    }

    get baseUrl(): string {
        return this.config.getOrThrow("BASE_URL");
    }

    get port(): number {
        return this.config.getOrThrow("PORT");
    }
}

[rajdip-b]

Can you give a reference usage of this API?

[jamesfrye420]

export class AppService {
  constructor(apiConfigService: ApiConfigService) {
    const db = apiConfigService.databaseurl
  }
}

Basically just use it as any other provider

[rajdip-b]

This looks cool! I think we can proceed with this pattern!

Also, in the docs, i saw a schema that can be passed to ConfigService using ConfigService<{...}>. Maybe we can use it too?

[kriptonian1]

We can just use zod to populate it to global scope, that will be much easier and clean

[jamesfrye420]

Can you share any docs or example usage?

[kriptonian1]

We can just do it like this file: src/env.server.ts

import { type TypeOf, z } from 'zod';

const zodEnv = z.object({
    PORT: z.string().refine((value) => /^\d+$/.test(value), {
        message: 'Expected a numeric string',
    }),
    PLATFORM_UI: z.string().url(),
    GOOGLE_CLIENT_ID: z.string(),
    GOOGLE_CLIENT_SECRET: z.string(),
    MONGO_URI: z.string().url(),

    JWT_SECRET: z.string(),

    SMTP_HOST: z.string().url(),
    SMTP_PORT: z.string().refine((value) => /^\d+$/.test(value), {
        message: 'Expected a numeric string',
    }),
    EMAIL: z.string().email(),
    EMAIL_APP_PASSWORD: z.string(),
});

declare global {
    // eslint-disable-next-line @typescript-eslint/no-namespace
    namespace NodeJS {
        interface ProcessEnv extends TypeOf<typeof zodEnv> {}
    }
}

try {
    zodEnv.parse(process.env);
} catch (error) {
    if (error instanceof z.ZodError) {
        const { fieldErrors } = error.flatten();
        const errorMessage = Object.entries(fieldErrors)
            .map(([field, errors]) =>
                errors != null ? `${field}: ${errors.join(', ')}` : field
            )
            .join('\n  ');
        throw new Error(
            'Missing environment variables: \n  ' +
                errorMessage +
                '\n  Please check your .env file.'
        );
    }
}

[jamesfrye420]

Since we are already using @nestjs/config module in my opinion it is just better to pass the zod schema to the nestjs validator.

declare global {
  // eslint-disable-next-line @typescript-eslint/no-namespace
   namespace NodeJS {
       interface ProcessEnv extends TypeOf<typeof zodEnv> {}
   }
}

This is actually a much simpler solution and I would strongly recommend implementing this as of now. The issue is in the original comment @rajdip-b wanted a way to not write process.env over and over again.

Now that I think about it, we can just re-export process.env under another const.

[kriptonian1]

I am not a fan of too much abstraction. It makes the code hard to debug.

[kriptonian1]

It feels very unnecessary to abstract process.env. Rather, what we can talk about it is how can we make more type safe.

[rajdip-b]

It feels very unnecessary to abstract process.env. Rather, what we can talk about it is how can we make more type safe.

Agreed on this front though @jamesfrye420. We just want these things:

• Make sure of the type safety
• Allow inference

[jamesfrye420]

In that case we don't need the solution I porposed since it adds too much complexity. We can shift to that once we start going crazy with the env vars (manipulating them, having custom logic etc.)

Just to get type safety and inference, the solution provided by @kriptonian1 works completely fine.

[rajdip-b]

Assigned this to your @jamesfrye420!

[jamesfrye420]

This issue requires 2 files,

env.validation.ts and env.d.ts

How should I structure this? Where should I put these files?

[rajdip-b]

I think you can use the common module for this. You can create a subfolder called env or environment and add those files in there.

[jamesfrye420]

Looking at the .env.example are there any mandatory variables as of now?

[jamesfrye420]

As of now the zod schema throws error whenever a Variable is missing from the .env file.

The problem is, if the variable is defined such that

DATABASE_URL=

It is passed to zod as DATABASE_URL: '' [an empty string].

By default all the variables in zod schema is required but zod valdiates empty string.

To overcome this we can set a .min() property which to the variables which are absolutely required and use .optional() property for the variables which can be missing from the .env files ?

On hindsight, if a variable is not required, it might be possible that the .env omits them completely so I think adding .optional() regardless makes sense

[rajdip-b]

Yeap adding optional will do the trick. And i thing keeping just .string() is equivalent to required.

Additionally, these are the required env vars:

• DATABASE_URL
• ADMIN_EMAIL
• SMTP details
• JWT_SECRET
• REDIS_URL

[jamesfrye420]

As I mentioned, .string() is equivalent to required but problem occurs when the variable is set to empty in the .env file. Since empty string passes the validation. To overcome this a .min() property needs to be set to ensure the strings are non empty and has value.