ctlstore icon indicating copy to clipboard operation
ctlstore copied to clipboard

[CONFIG-325] Increase ldb download speeds

Open zhou-hongyu opened this issue 1 year ago • 3 comments

  • Replace aws-sdk-go with aws-sdk-go-v2
  • Use concurrent downloading mechanism to download s3 snapshot with hardcoded concurrency configuration
  • replace golang's gzip with klauspost/gizp

zhou-hongyu avatar Aug 03 '23 22:08 zhou-hongyu

I know I may have asked this in our last 1-on-1, but do you have any benchmark / at least a snapshot of time of before / after this change? Even if you didn't profile the whole thing? If not should a separate PR be sent out first with just the metrics and then the change so the rough difference can be measured and observed?

asaf-erlich avatar Aug 29 '23 15:08 asaf-erlich

yes totally, the plan will be deploying to a ctlstore-beta node and validate the downloading speed via the dashboard I will circle back on this

zhou-hongyu avatar Aug 29 '23 15:08 zhou-hongyu

yes totally, the plan will be deploying to a ctlstore-beta node and validate the downloading speed via the dashboard I will circle back on this

Sounds good

asaf-erlich avatar Aug 29 '23 19:08 asaf-erlich