gosdk icon indicating copy to clipboard operation
gosdk copied to clipboard

Published 20 hours ago • verified publisher icon 0chain

refactor download file

Open cnlangzi opened this issue 3 years ago • 10 comments

  • download fragments by chunks in parallel requests (not Chunk on Protocols). It is https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests
  • download fragments in parallel requests from blobbers
  • resume a download of a file that has been partially transferred.
  • still need to support video-streaming . please check more detail with @moldis if there is any question. it is used on local or web video player
  • pipe stream support (output bytes to io.Writer instead of file directly)
  • support to reconstruct data on local with minimal fragments instead of download all fragments if it is possible . check here for detail

cnlangzi avatar Dec 03 '21 21:12 cnlangzi

some bugs need to be fixed too

  • folder is not able to download. we need to get file list , and download them one by one
  • please cancel download task if it is not able to download. eg auth ticket is incorrect or expired

cnlangzi avatar Dec 09 '21 07:12 cnlangzi

@cnlangzi Not clear abound point #1. The ranged request has been implemented in the oldest algorithm, are you going to reimplement it?

moldis avatar Dec 11 '21 16:12 moldis

@cnlangzi Not clear abound point #1. The ranged request has been implemented in the oldest algorithm, are you going to reimplement it?

yes, it should work in parallel requests too.

cnlangzi avatar Dec 12 '21 13:12 cnlangzi

Task taken. Will start on Sunday

lpoli avatar Dec 15 '21 01:12 lpoli

@lpoli , please double check ReadMarker. it seem be not designed for parallel requests.

cnlangzi avatar Dec 17 '21 14:12 cnlangzi

  1. Download fragments by chunks; So this is not supported by blobber yet? Please also explain about nature of parallel requests

  2. What support for video-streaming?

  3. Pipe stream is fine. Understood.

  4. I checked Reed Solomon docs. It is to reconstruct missing shards. So you mean we need this for repairing data, right?

@cnlangzi @moldis @guruhubb

lpoli avatar Dec 23 '21 17:12 lpoli

  1. Download fragments by chunks; So this is not supported by blobber yet? Please also explain about nature of parallel requests
  2. What support for video-streaming?
  3. Pipe stream is fine. Understood.
  4. I checked Reed Solomon docs. It is to reconstruct missing shards. So you mean we need this for repairing data, right?

@cnlangzi @moldis @guruhubb

re 1: ignore parallel requests in a blobber download. Because parallel requests is not supported on ReadMarker. re 2: https://docs.0chain.net/0chain/zbox-cli#video-streaming re 4: for now, all fragments are downloaded. but it is unnecessary. for example, we can only download 4 fragments only, and reconstruct original file if an allocation is 4 data + 2 parity.

cnlangzi avatar Dec 24 '21 23:12 cnlangzi

Readmarker doesn't support parallel requests. Are you suggesting it is due to ReadCounter?

Regarding video-streaming; I am still not sure what support is required?

lpoli avatar Dec 27 '21 10:12 lpoli

https://docs.0chain.net/0chain/zbox-cli#video-streaming

re ReadMarker

ReadMarker is verified one by one like WriteMarker. it is great if it is supported. please double check code

re video-streaming

please check it on https://docs.0chain.net/0chain/zbox-cli#video-streaming. I don't know other documentation too. it seems is implemented by @moldis ? if yes, try to ping him for more detail

cnlangzi avatar Dec 27 '21 12:12 cnlangzi

@moldis said they are using block download.

Now it seems only work is to pipe download stream to some writer/reader.

lpoli avatar Dec 27 '21 14:12 lpoli