storj
storj copied to clipboard
Reduce size satellite segment ID we are returning with begin segment response
Currently with metainfo BeginSegment
request we are returning lots of data. This affects our egress cost. We need to reduce size of response from this endpoint. First "quick" thing we can do is to reduce size of order limit we are putting into internal structure satellite segment ID. This is protobuf structure that satellite is serializing as SegmentID
that is used by libuplink. Currently we are putting into this structure all order limits that we also include directly into response. Each order limit can take around 300b and we are generating 110 order limit. 33000b and its just part of response.
Those order limits that are put into satellite segment ID doesn't have to me so large. We use data data later when segment is committed (metainfo CommitSegment
) to validate data and we are using only some of fields.
What we need to do:
- create custom structure with only fields that we need for validation (to replace
OriginalOrderLimits []*pb.AddressedOrderLimit
) - fill satellite segment ID new structure
- adjust places where validation were using OriginalOrderLimits from satellite segment ID to use new structure
- keep in mind that we cannot remove old field and code right away as during deployment mix of old and new API pods can cause errors
- we need to add new struct to satellite segment ID
- fill it and write new validation code that its using this struct
- still support old field
- deploy
- cleanup old code
Satellite segment ID: https://github.com/storj/storj/blob/main/satellite/internalpb/metainfo_sat.proto
https://github.com/storj/storj/blob/main/satellite/metainfo/endpoint_segment.go#L151