storj icon indicating copy to clipboard operation
storj copied to clipboard

Reduce size satellite segment ID we are returning with begin segment response

Open mniewrzal opened this issue 5 months ago • 1 comments

Currently with metainfo BeginSegment request we are returning lots of data. This affects our egress cost. We need to reduce size of response from this endpoint. First "quick" thing we can do is to reduce size of order limit we are putting into internal structure satellite segment ID. This is protobuf structure that satellite is serializing as SegmentID that is used by libuplink. Currently we are putting into this structure all order limits that we also include directly into response. Each order limit can take around 300b and we are generating 110 order limit. 33000b and its just part of response. Those order limits that are put into satellite segment ID doesn't have to me so large. We use data data later when segment is committed (metainfo CommitSegment) to validate data and we are using only some of fields.

What we need to do:

  • create custom structure with only fields that we need for validation (to replace OriginalOrderLimits []*pb.AddressedOrderLimit)
  • fill satellite segment ID new structure
  • adjust places where validation were using OriginalOrderLimits from satellite segment ID to use new structure
  • keep in mind that we cannot remove old field and code right away as during deployment mix of old and new API pods can cause errors
    • we need to add new struct to satellite segment ID
    • fill it and write new validation code that its using this struct
    • still support old field
    • deploy
    • cleanup old code

Satellite segment ID: https://github.com/storj/storj/blob/main/satellite/internalpb/metainfo_sat.proto

https://github.com/storj/storj/blob/main/satellite/metainfo/endpoint_segment.go#L151

mniewrzal avatar Sep 11 '24 08:09 mniewrzal