OpenSearch icon indicating copy to clipboard operation
OpenSearch copied to clipboard

[Searchable Snapshot] Implement the IndexInput classes for block-based storage

Open kotwanikunal opened this issue 3 years ago • 1 comments

Is your feature request related to a problem? Please describe.

  • The snapshot mechanisms in OpenSearch restore the entire index including the segment files for restoring an index
  • Searchable snapshot adds support for searching an index without downloading the entire files to disk
  • The above described protocol needs a new mechanism which will download parts of the file as required by the query

Describe the solution you'd like

  • The IndexInput interface can be utilized to create a virtual file mechanism which will download blocks of data as needed by the query
  • The underlying blocks will be cached onto the disk as files which can utilize the seek mechanism to load the appropriate block in memory using the IndexInput interfaces

Describe alternatives you've considered N/A

Additional context #3869

kotwanikunal avatar Jul 20 '22 16:07 kotwanikunal

POC feature commit: https://github.com/andrross/OpenSearch/pull/101

kotwanikunal avatar Aug 05 '22 17:08 kotwanikunal

I'm working on this.

aabukhalil avatar Sep 26 '22 17:09 aabukhalil