onionscan icon indicating copy to clipboard operation
onionscan copied to clipboard

Split Resources into Chunks for Storing

Open s-rah opened this issue 8 years ago • 0 comments

Currently we drop resources larger than 2MB because of limitations with the database - regardless of which backing store we end up with in the future, we are likely always going to need to chunk blobs.

We should

  • Configure a max object size for downloaded resources (set probably around 10MB for default)

  • Split resources smaller than this into <1MB chunks and store them in the DB in a way that can be put back together later e.g.

      resource:{
     url net.URL
     data []byte
     nextChunk int //id to resource chunk containing next part of the data
     }
    

s-rah avatar Oct 09 '16 05:10 s-rah