onionscan
onionscan copied to clipboard
Split Resources into Chunks for Storing
Currently we drop resources larger than 2MB because of limitations with the database - regardless of which backing store we end up with in the future, we are likely always going to need to chunk blobs.
We should
-
Configure a max object size for downloaded resources (set probably around 10MB for default)
-
Split resources smaller than this into <1MB chunks and store them in the DB in a way that can be put back together later e.g.
resource:{ url net.URL data []byte nextChunk int //id to resource chunk containing next part of the data }