Unable to decode base64 []: atob is not defined

[enzoferey]

Since https://github.com/upstash/upstash-redis/pull/198 I'm getting Unable to decode base64 []: atob is not defined warnings while running the library inside a Vercel API route.

I'm using automaticDeserialization, the feature might not be able compatible with it ?

Downgrading to version 1.13.1 does solve the issue, so it's clearly that commit that caused the issue.

[chronark]

@enzoferey

We have ci tests running on vercel and it worked there, I'll manually check this right now

what's your setup? Next.js on vercel I guess, and are you running a serverless or edge function?

[chronark]

I just tried this example with v1.14.0 and that works as expected. Can you give show me exactly what you are doing please?

[chronark]

I've also tried with nodejs directly, without a framework (added it as example here). That works too

[enzoferey]

Hey @chronark, sorry to drive you crazy on a Sunday without even providing much details 😅

Let me help you narrow this down. All the configuration I'm thinking may affect:

• @upstash/redis on v1.14.0 indeed.
• Next.js running on last version v12.3.1.
• Hosted on Vercel. Node.js version on the project is set to the 14.x option. Serverless function is running on Dublin, Ireland (dub1) region.

I can't post the exact code character by character, but this is exactly the same operations we do:

import { Redis } from "@upstash/redis";

const redis = new new Redis({
    url: UPSTASH_REDIS_REST_URL,
    token: UPSTASH_REDIS_REST_TOKEN,
    automaticDeserialization: false,
});

async function setNextValue(): Promise<void> {
    const [currentValue, someAsyncValueFromApi] = await Promise.all([
        redis.get("MY_KEY"),
        getAsyncValue(),
    ]);

    const nextValue = 
        currentValue !== null
            ? Math.max(parseInt(currentValue, 10) + 1, someAsyncValueFromApi)
            : someAsyncValueFromApi;

    await redis.set("MY_KEY", nextValue.toString());
}

This setNextValue is called frequently and most of the times called in parallel by several lambda functions invocations. We are protecting access to the key via the pattern described at https://redis.io/commands/setnx/ (not the Redlock multi-instance version, but the one described as the "old pattern"). That being said, I do not think this implementation detail affects the atob issue I have faced.

When debugging this issue, I logged the values of the equivalent of currentValue, someAsyncValueFromApi, and their types at the different steps of the function. Whenever the atob issue would pop, the returned value by redis.get would be an empty string, which parses to NaN and then the whole cycle would break.

Based on all my observations, I believe it's an issue about the base64 decoding implementation via atob and the Vercel lambda function environment.

I have not tried to setup a blank project and reduce the code around this logic, but I can spend time on it in the upcoming days if necessary.

[chronark]

👍 thanks, that helps I'll try to reproduce and debug this monday morning. Sounds like I missed an edge case and I can fix it fast. One more question: are you storing any special characters?

[enzoferey]

No, no special characters. It's just storing and retrieving an increasing integer number (to be more concrete, a nonce value). That's why I'm just using toString and then parseInt.

[chronark]

Tried reproducing but it works for me, can you double check please?

I created a completely fresh nextjs project

npx create-next-app@latest --ts --use-pnpm examples/216
cd examples/216
pnpm add @upstash/redis@latest

Then created the api route

// /pages/api/hello.ts
import type { NextApiRequest, NextApiResponse } from 'next'

import { Redis } from '@upstash/redis';

const redis = Redis.fromEnv({
  automaticDeserialization: false,
});

export default async function handler(
  req: NextApiRequest,
  res: NextApiResponse
) {
  try {
    const key = "enzoferey/216"

    const currentValue = await redis.get<string>(key)
    const asyncValueFromApi = 20

    const nextValue = currentValue !== null
      ? parseInt(currentValue, 10) + 1
      : asyncValueFromApi

    await redis.set(key, nextValue.toString())
    res.json({ currentValue, nextValue })
  }
  catch (e) {
    const err = e as Error
    console.error(err.message)
    res.status(500).send(err.message)
  }
}

It's running on vercel

Am I missing something?

[enzoferey]

Hey, thanks for taking the time. I have pinged many times the API route you setup and I cannot replicate neither. Does your Vercel project run on Node.js 14.x ?

Also, could the key be the where the decoding issue happens ? It looks something like this "nonce.LONG_HEXADECIMAL_ID_STRING".

[chronark]

Aaaah, I missed that detail. atob was introduced to node in v16 source

I redeployed the app with v14 and am getting the same error now.

Couple of solutions here:

1. Upgrade your app to node v16.
2. Add this polyfill to your api route

if (typeof atob === 'undefined') {
  global.atob = function (b64: string) {
    return Buffer.from(b64, 'base64').toString('utf-8');
  };
}

const redis = ....

3. Wait for me to fix this in the sdk (can't promise it this week unfortunately)

[enzoferey]

atob was introduced to node in v16 source

Right 😄 I should have checked this in the first place...

Thank you for looking into it. We have been using the downgraded version (v1.13.1) as we don't require any feature post that version and it works just fine for now. No hurries in getting the fix on the SDK, we will upgrade to Node v16 soon anyways.

[chronark]

👍 I'll leave this open until I merge the fix thanks for the report

[BurakGur]

if (typeof atob === 'undefined') {
  global.atob = function (b64: string) {
    return Buffer.from(b64, 'base64').toString('utf-8');
  };
}

I use this one for react native:

if (!global.atob) {
  global.atob = function atob(input) {
    var keyStr = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=';
    var output = '';
    var chr1, chr2, chr3;
    var enc1, enc2, enc3, enc4;
    var i = 0;
    input = input.replace(/[^A-Za-z0-9\+\/\=]/g, '');
    do {
      enc1 = keyStr.indexOf(input.charAt(i++));
      enc2 = keyStr.indexOf(input.charAt(i++));
      enc3 = keyStr.indexOf(input.charAt(i++));
      enc4 = keyStr.indexOf(input.charAt(i++));
      chr1 = (enc1 << 2) | (enc2 >> 4);
      chr2 = ((enc2 & 15) << 4) | (enc3 >> 2);
      chr3 = ((enc3 & 3) << 6) | enc4;
      output = output + String.fromCharCode(chr1);
      if (enc3 !== 64) {
        output = output + String.fromCharCode(chr2);
      }
      if (enc4 !== 64) {
        output = output + String.fromCharCode(chr3);
      }
    } while (i < input.length);
    return output;
  };
}