square
Representing a ByteString as a Source
There have been a couple times were I have wanted to pass a ByteString to another function as a Source. Currently I usually just write the ByteString into a Buffer and pass that to the function as a Source. However this completely duplicates the ByteString and could be a problem if the ByteString is significantly large.
This could be an extension function in Kotlin like ByteString.source() and exposed to Java as Okio.source(ByteString). Is this something you could see being useful for other projects like Wire, Moshi and/or OkHttp? If so, I can move forward with a PR.
How did you get the ByteString? If it's a segmented one (obtained from readByteString) it actually won't copy.
In general, I've had this use case at least 10 times over the years. Have you observed performance problems from this?
My use case is downloading blobs which looks like this.
suspend fun BlobAsyncClient.downloadByteString(): ByteString {
val buffer = Buffer()
download().asFlow().collect { buffer.write(it) }
return buffer.readByteString()
}
However, some of these blobs are compressed and others are not. So depending on the file name I need to un-gzip them which becomes a little more complicated.
val buffer = Buffer()
buffer.write(blob.downloadByteString())
val source: BufferedSource = if ("gz" in blobName) (buffer as Source).gzip().buffer()
else buffer
It would be nice if I could just use something like this:
blob.downloadByteString()
.source()
.let { if ("gz" in blobName) it.gzip() else it }
.buffer()
Yes, I benefit here because readByteString() will optimize a SegmentedByteString if significantly large. So for my particular use case I don't really have any performance or memory concerns, but this also isn't the first time I've had need for a ByteString source either.
Or maybe this is a need for a unGzip() (poor name) or similar function on ByteString similar to the hash functions? But a source() function would allow transforming Sources like HashingSource, DeflateSource, or GzipSource to operate on the ByteString efficiently. I think source() could be a good building block for composability.
Something which came to mind is this might be resolved by #680 which would add Source/Sink analogs which supported positional reading and writing. ByteString could implement the positional reading type and an asSource() extension function on that super-type would be very natural.
I've been wanting to pick the Store idea back up again though I no longer have a use case for it. Are you still interested in a Store abstraction? I can revisit #680 if you are; except for a few naming things, I was pretty happy with the minimal API it added.
I expect our implementation of ByteString.source() will be similar to this:
fun ByteString.source() = Buffer().write(this)
The only way to get data out of a source is via a Buffer, and so it’s going into a buffer sooner or later. Once it’s in any buffer the copies between ’em are free.
