ipfs-log icon indicating copy to clipboard operation
ipfs-log copied to clipboard

Published 20 hours ago • verified publisher icon orbitdb-archive

Feat: improve memory usage / allow for in-memory storage of all entries to be optional (WIP)

Open mistakia opened this issue 4 years ago • 6 comments

⚠️ Note: This PR is a WIP and not ready for review. I'm in the process of cleaning up a fork that I have been using in a proof of concept application. My hope is to minimize the amount of breaking changes further.

Description

Currently, all entries (and all the data in it) are kept in memory. The goal of this PR is to make this optional in order to improve load times and performance on memory-limited environments when interacting with larger/longer log sets.

Related: https://github.com/orbitdb/ipfs-log/issues/136, https://github.com/orbitdb/ipfs-log/issues/203, https://github.com/orbitdb/ipfs-log/pull/239

Breaking Changes

The following getters and methods become async / return promises.

  • log.values
  • log.tails
  • log.tailHashes
  • log.get()
  • log.traverse()
  • log.iterator()
  • log.toString()
  • log.toSnapshot()

Benchmarking

Before wip

After wip

Tradeoffs

wip

TODO

  • [x] update tests
  • [ ] write new tests
  • [ ] update benchmarks / benchmarking / document performance tradeoffs
  • [x] adjust entryIndex to allow caching of all entries by default
  • [ ] document breaking changes
  • [x] update method comments
  • [x] squash / clean up commits
  • [x] remove forked orbit-db-io

mistakia avatar Apr 22 '20 15:04 mistakia

Excited about this if it turns out to work! Would you mind though making this a draft PR until it's ready?

aphelionz avatar May 07 '20 01:05 aphelionz

How are you feeling about this one @mistakia ? :)

aphelionz avatar Aug 21 '20 00:08 aphelionz

Feeling like I need to get back in the game! I plan on coming back to this in the not so distant future.

mistakia avatar Aug 21 '20 01:08 mistakia

Ok, I'll leave it open then :+1:

aphelionz avatar Aug 21 '20 02:08 aphelionz

@mistakia the not too distant future is upon us :D hope you are happy + healthy in these trying times.

If you want to hop back in I can totally support you. Did you see I finally merged #307 ?

aphelionz avatar Dec 11 '20 21:12 aphelionz

Quick status update of what I think is left. Merge the following into ipfs-log master:

  • [x] https://github.com/orbitdb/benchmark-runner/pull/17
  • [ ] https://github.com/orbitdb/benchmark-runner/issues/26

After these two I can dig deeper in the benchmarks to make improvements (i.e. analyze various LRU libs)

  • [x] update log.spec tests
  • [ ] what to do with errors when loading an entry from a multihash
  • [ ] document breaking changes
  • [x] update docs

mistakia avatar Jan 20 '21 19:01 mistakia