joystream icon indicating copy to clipboard operation
joystream copied to clipboard

Colossus state api re-factor

Open mnaamani opened this issue 1 year ago • 1 comments

Evaluate need for state API endpoints:

  • /state/data-objects
    • Returns JSON array of all local data objects stored by node.
    • Was previously used by v3.8.x nodes to determine if a node should try to fetch an object from remote node. But we switched using simpler "dumb" approach of just trying all nodes where object is expected to be available. Mainly because the response size was on order of several megabytes, and seemed wasteful. It would now only serve as a "diagnostic" endpoint.
  • /state/bags/{bagId}/data-objects
    • This is similar to /state/data-objects but performs a gql query on each request and a potentially slow _.intersection() on large array.
  • /state/data
    • Returns usage size of uploads,temp,pending folders
    • On each request the stats are re-computed (lots of async fs i/o) seems wasteful and results should be cached

mnaamani avatar Jan 24 '24 06:01 mnaamani