Shwap

[Wondertan]

Prototype of the new storage for edses - edsfile and optimized roundtrip less ipldv2 protocol

TODO:

• edsfile
  • [x] Blockstore over EDSFile
  • [x] Support for Col sample reads
  • [x] ODS only mode
    • [ ] ~Support for Axis, Share and ShareWithProof~
  • [x] Measure size diff
  • [x] Delete EDS mode
  • [x] GetSharesByNamespace (assigned to @walldiss)
  • [ ] File streaming (Write / Read)
  • [ ] Concurrency safety for file
• edsStore
  • [ ] Base implementation
  • [ ] Caching
  • [ ] Migrate shrex server to new storage
  • [ ] Store getter
• ipldv2
  • [x] Define Sample type and refactor the protocol and the Hasher
  • [x] Migrate to proto
  • [x] Row Sampling
  • [ ] ~Versioning and backwards compatibility with old protocol~
  • [x] Use heights instead of DataHash to identify block
  • [x] Hash requested Axis roots to shorten requests
  • [ ] Retriever over v2 (IPLDv2 reconstruction getter) (assigned to @walldiss)
  • [ ] Metrics
  • [x] IPLDv2 row based getter
    • [x] Getter should support requesting batch of shares by idx at once
    • [x] Store only samples by default
    • [x] Bitswap Sessions
  • [ ] ~IPLDv2 Server over LN storage (blockservice)~ Will work as is
  • [ ] Migrate actual sampling logic over GetShares #2984

Stretch Goals:

• Range of shares Multihash(by extending SampleID with additional uint16 that defines the end idx of range)
• Blob requests. BlobID{DataID, Comm} -> Blob{BlobShares, ProofsToSubroot}
• Requesting proofs only for every request
• Reconsider DHT discovery.

[Wondertan]

would deprecate the gymnastics of reading from the car file done in https://github.com/celestiaorg/celestia-node/pull/2638

@derrandz, very true

[codecov-commenter]

Codecov Report

Merging #2675 (2baea7d) into main (303db05) will increase coverage by 0.22%. Report is 9 commits behind head on main. The diff coverage is 54.48%.

@@            Coverage Diff             @@
##             main    #2675      +/-   ##
==========================================
+ Coverage   51.79%   52.02%   +0.22%     
==========================================
  Files         163      173      +10     
  Lines       10854    11596     +742     
==========================================
+ Hits         5622     6033     +411     
- Misses       4738     4997     +259     
- Partials      494      566      +72     

Files	Coverage Δ
share/eds/file_store.go	0.00% <0.00%> (ø)
share/ipldv2/axis_sample_hasher.go	56.00% <56.00%> (ø)
share/ipldv2/ipldv2.go	60.71% <60.71%> (ø)
share/ipldv2/share_sample_hasher.go	56.00% <56.00%> (ø)
share/eds/file_header.go	51.16% <51.16%> (ø)
share/ipldv2/axis_sample_id.go	63.07% <63.07%> (ø)
share/ipldv2/share_sample_id.go	64.17% <64.17%> (ø)
share/ipldv2/share_sample.go	71.17% <71.17%> (ø)
share/eds/ods_file.go	15.38% <15.38%> (ø)
share/ipldv2/axis_sample.go	62.36% <62.36%> (ø)
... and 2 more

... and 15 files with indirect coverage changes

[Wondertan]

Initial file size measurements

EDS 256:
	NEW: ods_file: 8388640(-9.36x)
	NEW: eds_file: 33554464(-2.34x)
	OLD: car_file: 78618134

EDS 128
	NEW: ods_file: 2097184(-9.36x)
	NEW: eds_file: 8388640(-2.34x)
	OLD: car_file: 19631894

[fl0rek]

I don't think it's necessary but other protobuf files specify the namespace,so we expected this one too

On Sun, 31 Dec 2023, 2:26 am Hlib Kanunnikov, @.***> wrote:

@.**** commented on this pull request.

In share/shwap/pb/shwap_pb.proto https://github.com/celestiaorg/celestia-node/pull/2675#discussion_r1438769871 :

@@ -0,0 +1,26 @@ +syntax = "proto3";

It's not necessary from my understanding

— Reply to this email directly, view it on GitHub https://github.com/celestiaorg/celestia-node/pull/2675#discussion_r1438769871, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACF3U6BLASXRHTIRAXGG4KTYMC5LPAVCNFSM6AAAAAA4PNTBF2VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTOOJZGYYDCMJRGY . You are receiving this because you commented.Message ID: @.***>

[Wondertan]

We merged #3675, so can close this one