@vercel/kv Docker image for local development

[sebpalluel]

I'm really interested by your SDK but i've noticed that you need to use an URL and Token from Vercel and i see no option to connect a Redis instance directly. I'm used to work with local containers for postgres and redis through docker-compose, which is best for many reason including for tests and the ability to work even with a degraded internet connection. Do you plan to release a docker image for @vercel/kv or at least a way to connect to a local redis instance ?

[coreh]

Since Vercel KV is based on the @upstash/redis library, you can use Serverless Redis HTTP (SRH) to run a local HTTP endpoint for development. (See: https://docs.upstash.com/redis/sdks/javascriptsdk/developing-or-testing)

This is my current docker-compose.yml setup:

version: '3.1'

services:
  redis:
    image: redis
    ports:
      - '6379:6379'

  serverless-redis-http:
    ports:
      - '8079:80'
    image: hiett/serverless-redis-http:latest
    environment:
      SRH_MODE: env
      SRH_TOKEN: example_token
      SRH_CONNECTION_STRING: 'redis://redis:6379' # Using `redis` hostname since they're in the same Docker network.

You can then connect with the following environment variables:

KV_REST_API_URL="http://localhost:8079"
KV_REST_API_TOKEN="example_token"

[sebpalluel]

Since Vercel KV is based on the @upstash/redis library, you can use Serverless Redis HTTP (SRH) to run a local HTTP endpoint for development. (See: https://docs.upstash.com/redis/sdks/javascriptsdk/developing-or-testing)

This is my current docker-compose.yml setup:

version: '3.1'

services:
  redis:
    image: redis
    ports:
      - '6379:6379'

  serverless-redis-http:
    ports:
      - '8079:80'
    image: hiett/serverless-redis-http:latest
    environment:
      SRH_MODE: env
      SRH_TOKEN: example_token
      SRH_CONNECTION_STRING: 'redis://redis:6379' # Using `redis` hostname since they're in the same Docker network.

You can then connect with the following environment variables:

KV_REST_API_URL="http://localhost:8079"
KV_REST_API_TOKEN="example_token"

Thanks for your solution, I will try it out ! That would be good to have it in the doc as it was clearly an issue that kept me out of using this service on the first place.

[vvo]

Hey there, we're looking at adding pages for both postgres and kv about local development, will keep you posted

[tweeres04]

Since Vercel KV is based on the @upstash/redis library, you can use Serverless Redis HTTP (SRH) to run a local HTTP endpoint for development. (See: https://docs.upstash.com/redis/sdks/javascriptsdk/developing-or-testing)

Upstash developing or testing page has moved to: https://upstash.com/docs/oss/sdks/ts/redis/developing

[jpchavat]

Hey there, we're looking at adding pages for both postgres and kv about local development, will keep you posted

any progress, @vvo ?

[0x33dm]

Hey there, we're looking at adding pages for both postgres and kv about local development, will keep you posted

any progress, @vvo ?

Not being able to work offline is a deal breaker.

Being able to run tests offline is quite an important when developing an API.

[kerkkoh]

@0x33dm The docker container solution by @coreh above works as of 2024-04-20. Simply download Docker desktop, and in your project directory root, create a file called docker-compose.yml and copy & paste the contents:

version: '3.1'

services:
  # This is the local Redis database that will be running on your computer; Within docker it has the hostname `redis` and it exposes the port 6379 to your pc and other containers
  redis:
    image: redis
    ports:
      - '6379:6379'

  # This is the serverless router that allows the Vercel KV libraries to send queries to the redis server above (redis:6379)
  # It uses the port 80 internally, so it is mapped to an arbitrary port 8079, so that it doesn't conflict with your potential web app on port 80
  # Think of this as local Vercel KV, and you could even rename this to local-vercel-kv if that helps you to understand this structure
  serverless-redis-http:
    ports:
      - '8079:80'
    image: hiett/serverless-redis-http:latest
    environment:
      SRH_MODE: env
      SRH_TOKEN: example_token # This is your local Vercel KV instance's API token (KV_REST_API_TOKEN), and you can change it to anything or keep it as is for development
      SRH_CONNECTION_STRING: 'redis://redis:6379' # Using `redis` hostname since they're in the same Docker network.

Now when you have Docker Desktop (or other version of Docker) installed and running, simply open up a command line (bash/cmd/terminal) and navigate to the directory where you placed the docker-compose.yml file, and execute the following command:

docker-compose up

Now you should see Docker downloading the needed images, and ultimately the local Vercel KV instance starts running.

Finally, within your project of choice, expose the 2 system variables required for the @vercel/kv library to connect to your local KV instance in a .env file for example like above:

KV_REST_API_URL="http://localhost:8079"
KV_REST_API_TOKEN="example_token"

Hopefully this further explanation helps you out! This docker solution is the best one right now (and probably in the future) as you can deploy this locally on any operating system. If you have any questions, let me know.

As @tweeres04 kindly pointed out, you can also find a similar setup that will work with @vercel/kv in the Upstash documentation https://upstash.com/docs/oss/sdks/ts/redis/developing.

Vercel KV itself is simply a dumbed down wrapped version of Upstash, so generally anything that applies to Upstash will apply to Vercel KV. If you want to find out more, I'd suggest looking at what you get with Upstash directly instead of using Vercel KV as a wrapper. Disclaimer: I have nothing against Vercel KV, and use it in production

[0x33dm]

@kerkkoh thanks for the info, that makes sense.

Having to run a docker compose locally for running tests seems a bit over the top when running simple tests locally tough.

I would prefer having a simple redis-mock for my tests locally instead of requiring developers to run an entire docker stack.

I believe this wouldn't work for KV tough as from my understanding KV does add a bit of extra sauce on top of the simple redis server ( and that's why we need hiett/serverless-redis-http:latest on the docker compose ) right?

[huksley]

Can't we just have a "local dev" mode in kv, when it stores in filesystem? I.e. VERCEL_KV_LOCAL=1 and no need to use heavy weight docker container...

[kerkkoh]

@0x33dm I suggest looking at what the serverless-redis-http docker container is running, which is an HTTP proxy for a Redis server. It's meant to catch any requests to an HTTP API, which is placed in front of Redis, which is what Upstash/KV do in the cloud. See https://github.com/hiett/serverless-redis-http/tree/master. Btw, they have instructions for how to set that docker container to run with e.g. GitHub jobs, which you can apply to other CD/CI services as well.

You can create a similar setup as serverless-redis-http in your language of choosing (or use their Elixir) and repackage that into redis-mock, which will catch the HTTP requests that Vercel KV/Upstash client libraries such as @vercel/kv send to the actual Redis instance, and your implementation could simply catch & kill those requests, or do what @huksley said, and write them into a file. In this case, you are lacking the actual Redis implementation, so your tests will have to consider that the @vercel/kv calls would go nowhere (i.e. your calls do not affect any database) unless you want to implement Redis again but as a flat-file database. For this approach, I suggest looking for packages that have implemented a local Redis in JS like redis-js.

Equally reasonable approach is to contribute to open-source, and write support into this repository for @huksley's VERCEL_KV_LOCAL=1, and simply cancel all the requests that the package is sending. I haven't looked at the code personally, but I imagine you can override some HTTP client with a mocked client that returns nothing but also doesn't fail your tests.

Edit: Just looked at the package and... yeah it's just a wrapper, just look at how little action there is in the code. 😂 This package does absolutely nothing, and that's the reason why nothing has changed. Just skip the use of the @vercel/kv package altogether, and just switch to @upstash/redis -- And this means also opening up an issue about this at upstash/upstash-redis

The above is what maintainers of this package should consider, but as this issue has been open for over a year, and everyone has adopted the local Vercel KV/Upstash instance through the docker approach, I doubt we will see anything change at least in this package by its maintainers. Vercel is a profit-seeking corporation, and they want you to create a second test KV database so that they can bill you for that instance as well. If you adopt their approach, I suggest using Upstash directly (same API, same databases, but lower costs).

Personally, I just run the local Docker images, allocate them very little memory & CPU, and let them automatically restart whenever my Docker does. This allows for much more thorough testing, as the requests your @vercel/kv library sends will make changes to an in-memory DB, Redis.

[0x33dm]

@kerkkoh I'm not seeing any advantages on switching from "regular Redis" to KV, am I missing something?

I'm just using Redis for simple cache and having to deal with all this extra setup mentioned to use KV instead seems like a lot more work than setting my Redis cache on a regular Redis server and running my tests using my usual Redis test library. ( again... am I missing something? )

[kerkkoh]

@0x33dm There are essentially no reasons to switch from Redis to KV, especially if you are just using it only locally. The appeal of Vercel KV is often that it offers persistence for the Redis instance, meaning that they use the persistence features of Redis, which write your data to a file as well as memory, and hot-swap files to memory when needed. If you don't need this feature, you likely don't need a Vercel KV instance, but as you correctly indicated, you just need a simple Redis cache.

"Vercel KV" is just a buzzword deployed by Vercel to refer to Upstash Redis, which they are re-selling to Vercel customers with the benefit being that you can manage the Upstash Redis instance through Vercel, and your projects automatically can have the connection details injected into your project's environment variables. Vercel however adds so much to the costs, has a subset of Upstash features available, and is overall a much worse database option than Upstash or just simple Redis hosted somewhere (in my opinion).

Upstash Redis on the other hand is much cheaper than Vercel's of course, and essentially scales to near zero costs because they run serverless Redis instances, which are persistent (you can use it like MongoDB or an SQL database). They also offer a wide set of analytics and other features that are disabled on Vercel KV instances. Upstash bills you per usage, so no more Hobby, Pro, and Enterprise plans to worry about -- you will only pay for what you use. They also offer edge-optimized options, so your queries are snappy and fast for customers connecting to it worldwide.

I've tried to indicate which parts are my own opinion, but the facts on the offerings of each service should be on point.