experimental raw support tracker

[pi0]

Update: We should consider adopting undio for standard raw types.

---

Initial issue: #21. Tracking experimental getItemRaw/setItemRaw support added in https://github.com/unjs/unstorage/pull/141. Please comment on any particular issues and ideas with the current implementation.

Experimental limitations:

• Default serializer is only supported in environments supporting Buffer. (Node.js and Nitro/Nuxt server in any environment)
• Cannot combine string and raw operations on the same key
  • Default serializer only operates on base64: strings without json support
  • This makes issues with overlay and http backends when translating string operations
• Types are wide to any since we don't enforce any return value from drivers implementation.
• We might support stream responses too

[ashutosh887]

@pi0 Does this need some refractor?

[xzilja]

Was attempting to store CryptoKey type, but got an error as it didn't satisfy allowed types (believe some of the listed ones were ArrayBuffer, UTF...)

[pi0]

@0xAsimetriq would you please make a reproduction if have an issue with storing raw data? 🙏🏼

[dnl13]

Hello,

Since this topic is also flagged as a discussion, I have a short question. My goal: I would like to cache/store WASM models and binaries in indexedDB. Since these models are intended for client-side use only and my NuxtJS Composable is also only called on the client-side, I have some questions about setItemRaw.

I have compared both setItemRaw() and set() from idb-keyval. With set(), I can easily store my Uint8Array in indexedDB on the client-side without any issues.

However, I get the error "ReferenceError: Buffer is not defined" due to the serializer in setItemRaw, which is understandable because Buffer is not available on the client-side.

My question is whether we can simply add set and get from indexedDB to our createStorage with the indexedDB-Driver to support these functions on the client-side, or why serialization is necessary at all (shouldn't developers be responsible for serializing data if it can't be stored in indexedDB in the desired format)? The term setItemRaw implies that the data is appended without any additional processing, so perhaps the wording isn't ideal.

I am relatively new to this topic and unfortunately do not have very deep knowledge about it. I just wanted to share my observation and ask how I could store my Uint8Array with unstorage and the indexeddb driver without having to convert it to base64, which can be quite performance-intensive for a model of around > 350MB in size.

Attached is a short NuxtJS composable that I tried:

import { createStorage } from "unstorage";
import indexedDbDriver from "unstorage/drivers/indexedb";
import { get, set , createStore  } from 'idb-keyval';

export const useModelStore = async () => {

  const store = createStorage({
    driver: indexedDbDriver({ base: "app", dbName: 'model-store', storeName: 'model' }),
  })
  const idbKeyvalAdapter = createStore('keyval_model-store', 'keyval_models');

  const setModel = async (key) => {

    store.setItemRaw(key, new Uint8Array([1, 2, 3])); // ReferenceError: Buffer is not defined

    set(key, new Uint8Array([1, 2, 3]), idbKeyvalAdapter); // works
  }

  setModel("Modelname")
  ...
}

<script setup lang="ts">
onMounted(async () => {

  const { setModel  } = await useModelStore ();
  ...

For now, I will use idb-keyval for these models. I would greatly appreciate any tips on how to avoid this redundant dependency and how to solve the entire thing using unstorage. Thank you very much!