abstract-blob-store icon indicating copy to clipboard operation
abstract-blob-store copied to clipboard

is there a level-blob-store ?

Open jbenet opened this issue 9 years ago • 5 comments

is there a level-blob-store that allows using any level instance as a blob-store? i know that's very much not the goal of blob-store (big blobs instead of small values) but sometimes useful if you already have the data there and just want to read it with same interface

jbenet avatar Dec 16 '15 09:12 jbenet

I started one yesterday https://github.com/diasdavid/level-blob-store (will release when it complete, hopefully not later than today), since I didn't find any at the list: https://github.com/maxogden/abstract-blob-store#some-modules-that-use-this

daviddias avatar Dec 16 '15 10:12 daviddias

I wrote this a while back before abstract-blob-store was a thing but it actually shares the same interface, https://github.com/mafintosh/level-blobs

mafintosh avatar Dec 16 '15 11:12 mafintosh

ohh by 3 minutes xD https://github.com/diasdavid/level-blob-store

Nice, thank you @mafintosh, now we have two! :D Seems yours is already set in a way that the level storage is passed afterwards so that it runs in the browser and in Node.js, awesome!

daviddias avatar Dec 16 '15 11:12 daviddias

@diasdavid I dont wanna discourage use of a level blob store module, but it would be worth mentioning two things in your readme:

  • LevelDB doesn't support streaming file writes, so values must fit in memory.
  • In Google LevelDB compaction is notoriously inefficient in heavy write scenarios, so writing large quantities of data will likely result in lots of CPU thrashing.

max-mapper avatar Dec 16 '15 18:12 max-mapper

Thank you, @maxogden :) I'll add that to the README, it was one of the things I found, but fortunately for our scenario, levelDB will be most to store DHT records and we are in fact moving out of levelDB for that too, it is just to be able to support legacy repos.

daviddias avatar Dec 17 '15 05:12 daviddias