Missing API: zero-copy wrapping of ByteArray with Source.

[SPC-code]

There are several cases in binary parsing, where one needs to wrap a ByteArray with an interface like Source and read from it sequentially. It should be possible by using a Buffer with a single Segment, but Segment API is private.

There should probably be two versions of the API: read/write and read-only.

[whyoleg]

I think, that wrapping ByteArray (or ByteBuffer) or any other unsafe API for working with Segment will be considered in scope of #135

[joffrey-bion]

I would also love to see this added, and I would add that an API for turning ByteString into a Source directly would also be very useful, because we cannot publicly access the ByteString's backing byte array from outside this library.

[fzhinkin]

@joffrey-bion

we cannot publicly access the ByteString's backing byte array from outside this library.

In fact, you can write something like:

@file:OptIn(UnsafeByteStringApi::class)
import kotlinx.io.bytestring.*
import kotlinx.io.bytestring.unsafe.*

val str = ByteString(1, 2, 3)

UnsafeByteStringOperations.withByteArrayUnsafe(str) { arr -> 
     println("There are ${arr.distinct().size} distinct bytes inside!")
}

[joffrey-bion]

Thanks for this! I wasn't aware, but that looks quite promising. I will check if this solves my problem and report back.

[fzhinkin]

There was an intention to avoid explicitly mentioning the unsafe API, so nobody would find it until that API is really needed. But it seems like we made it too hidden. :)

[joffrey-bion]

I put my thoughts in https://github.com/Kotlin/kotlinx-io/issues/259, because I don't want to pollute this issue any further (it's related but different)

[Chuckame]

After reading the #135 and #311 , I'm not sure, is there something around wrapping a ByteArray as a Source or a Buffer to enable zero-copy ? Sorry if I misread. It would add so much kotlin power for avro4k, allowing me to only provide an API using kotlinx-io Source and Sink, and the users are free to wrap a source from a ByteArray, ByteBuffer, a file, network, etc

[fzhinkin]

@Chuckame please check https://github.com/Kotlin/kotlinx-io/blob/fd49af54f21703bedcf87082a4d8e5caa770c1fb/core/common/src/unsafe/UnsafeBufferOperations.kt#L37

[Chuckame]

Thanks, so is this open issue still relevant ?

[fzhinkin]

I'll close it once #334 will be merged

[Chuckame]

Oh sorry I just discovered that it's from a PR 😄 Will you release after this PR merged ? Or will you wait for the 5/5 PR merged ?

[fzhinkin]

The plan was to merge PRs "2/5" and "3/5", fix a few issues with segment pools and release it without "4/5" and "5/5" (which could be merged and released later).

[Chuckame]

Ok thanks.

Is there some official roadmap and milestones with releasing dates ? (not expecting exact vision, just to know if it's about days/week, or for the end of the year — or more)

In any case, I'll follow the PRs. It's to see if I can wait a bit to use kotlinx-io before releasing our v2.

[fzhinkin]

The unsafe API, at least aforementioned part, will be released before July the 15th.

[fzhinkin]

@Chuckame, the API was included in the recent 0.5.0 release.

[Chuckame]

3 days earlier, what a boss! Thanks, I'll test it 🚀