deno

deno copied to clipboard

ATM the local KV instances are always backed by an SQLite database.

There would be at least two use-cases where having a memory only backend would be interesting:

• Memcached/Redis alternative, this has the advantage that you get no dependencies and possible extra performance from the in-process memory. Plus, you get a uniform API for working with ephemeral or persistent storages.
• An interesting and possible more performant way to implement Web worker concurrency, this would be implemented with the semantics of the KV atomic operations.

Deno.openKv(":memory:"); :)

Deno.openKv(":memory:"); :)

Interesting, never though you would pass that straight to sqlite 👍🏻

Looks that you can do this shared in memory db and pass data from/to workers

if (!import.meta.url.endsWith("#worker")) {
  // Shared cache
  const kv = await Deno.openKv("file::memory:?cache=shared");
  await kv.set(["foo"], "bar");

  // Start worker
  const worker = new Worker(import.meta.url + "#worker", {
    type: "module",
  });

  // Listen to messages enqueued by worker
  kv.listenQueue((msg: unknown) => {
    console.log(msg);
  });
} else {
  // Shared cache
  const kv = await Deno.openKv("file::memory:?cache=shared");

  // Read from shared cache
  console.log(await kv.get(["foo"]));

  let count = 0;

  setInterval(async () => {
    await kv.enqueue({ msg: "Hello from worker", count: ++count });
  }, 1000);
}

Stressing this a bit doesn't go that well. Just changing to:

setInterval(async () => { await kv.enqueue({ msg: "Hello from worker", count: ++count }); }, 5);

will crash Deno

Platform: macos aarch64
Version: 1.38.3
Args: ["deno", "run", "-A", "--unstable", "worker.ts"]

thread '<unnamed>' panicked at /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/denokv_sqlite-0.2.1/lib.rs:215:15:
KV queue dequeue failed: database table is locked
stack backtrace:
  0:        0x1010d3f98 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h4bff038afc05d0ab
  1:        0x10053caec - core::fmt::write::he9fa43146d420946
  2:        0x1010a8824 - std::io::Write::write_fmt::hf4fd30c8192b15d9
  3:        0x1010d86c4 - std::sys_common::backtrace::print::h5d71f7b09493fc9b
  4:        0x1010d831c - std::panicking::default_hook::{{closure}}::hb75620628e95e648
  5:        0x1010d7f88 - std::panicking::default_hook::h4ac4acc64d21c4ad
  6:        0x10040ea54 - deno::setup_panic_hook::{{closure}}::h6bc9ac56a78a26f4
  7:        0x1010d8fb0 - std::panicking::rust_panic_with_hook::h81dc715fa5de32cf
  8:        0x1010d8d88 - std::panicking::begin_panic_handler::{{closure}}::h26c73eb75670321f
  9:        0x1010d8cf4 - std::sys_common::backtrace::__rust_end_short_backtrace::hef4773cf0f80485b
 10:        0x1010d8ce8 - _rust_begin_unwind
 11:        0x10053b004 - core::panicking::panic_fmt::h10bd3bfe5bb30d67
 12:        0x100ea67b8 - denokv_sqlite::sqlite_thread::{{closure}}::hb00c634db1831b2c
 13:        0x100ea2438 - std::sys_common::backtrace::__rust_begin_short_backtrace::hf24b61066522010f
 14:        0x100eac85c - core::ops::function::FnOnce::call_once{{vtable.shim}}::h69fc3aa8957ff612
 15:        0x1010dc100 - std::sys::unix::thread::Thread::new::thread_start::h616a123cfe4994dc
 16:        0x18827d034 - __pthread_joiner_wake

SQLite only supports one writer at a time per database file. If many threads and/or processes write the database at the same instant, will cause database table is locked！