Download a large file in batches

[roshni06]

Description of the Issue

I want to download all the files from Box and upload to AWS S3 bucket. I am looking into APIs for that and found i can do using below,

BoxFile file = new BoxFile(api, "id");
BoxFile.Info info = file.getInfo();

FileOutputStream stream = new FileOutputStream(info.getName());
file.download(stream);
stream.close();

https://developer.box.com/guides/downloads/file/

If I have a file of size 1TB, I want to download this in batches rather than entire 1TB as this would occupy a lot of memory. Is there an API that supports batch download?

Please let me know.

Thanks

[antusus]

Hello @roshni06,

we do not have API that allows you to download files from Box in batches. Maybe you can try and use range requests to create file parts and upload them to S3.

BoxFile file = new BoxFile(api, "file_id");
long size = file.getInfo().getSize();

int batchSize = 8;
long downloadedSize = 0L;
int part_number = 1;
while (downloadedSize < size) {
    OutputStream outputStream = new FileOutputStream("file_in_chunks_" + part_number);
    long range = Math.min(batchSize, size - downloadedSize);
    long from  = downloadedSize;
    long to  = from + range;
    System.out.println(String.format("Downloading from %d to %d of %s", from, to, size));
    file.downloadRange(outputStream, from, to);
    downloadedSize += range + 1;
    outputStream.close();
    part_number++;
}

[stale[bot]]

This issue has been automatically marked as stale because it has not been updated in the last 30 days. It will be closed if no further activity occurs within the next 7 days. Feel free to reach out or mention Box SDK team member for further help and resources if they are needed.

[stale[bot]]

This issue has been automatically closed due to maximum period of being stale. Thank you for your contribution to Box Java SDK and feel free to open another PR/issue at any time.