Stream serialization using multiplatform OKIO

[fab1an]

What is your use-case and why do you need this feature?

I have ported the Stream interface from google's gson to kotlin-multiplatform using okio.

Library: https://github.com/fab1an/kotlin-json-stream API-docs: https://fab1an.github.io/kotlin-json-stream/index.html

The API-documentation is not yet done, but everything works and is published on mavencentral, please give it a try.

It enables you to stream through json, using methods like

• startObject()/endObject()
• peek() == JsonToken.BEGIN_ARRAY
• nextInt()/nextBoolean()
• ≥

[JakeWharton]

There's a fair amount of allocations on the hot path. I would assume this is mostly due to porting a library that's char-based whereas Okio is byte-based, and then maybe not being extremely comfortable with Okio's very low level API. But you're on the right track!

I would consider looking at Moshi for inspiration. It's Gson 3.0 in everything but name by the people who maintained Gson 2.x for many years. It also operates on bytes and not chars, and is fundamentally Okio-based.

The main thing that I would want to see improved with your implementation is the removal of all ByteString creation in the read path. There's quite a lot currently. Okio has some low-level APIs for reading raw bytes which you can then match against rather than allocating ByteStrings to do range matching.

Moshi also just has a superior JsonReader and JsonWriter API that lets you do things like get access to values as raw sources and sinks, as well as the peek feature which lets you conditionally read ahead as much as you want to facilitate polymorphic deserializatiom or whatever you want. I think it would be a really nice improvement to include these advancements on Gson's versions.

[fab1an]

You are right. These bytestrings are all the same and can be easily factored out and instantiated statically once.

Also I didn't use Moshi because it is not multiplatform. My library is, it is available for native, Java, iOS and can be used in JS as well all while using kotlin common code.

[JakeWharton]

I'm saying you should port from Moshi rather than Gson because it has the superior design (between the two).

[fab1an]

@JakeWharton Please have a look at the code again, is that what you had in mind? I am working on a serialization-library that builds on top of this. I will have a look at Moshis new interface.

[fab1an]

I had a look at Moshi and I see what you are getting at. I don't think it makes sense for me to port my code to moshi's interface since I am not that deep into the topic, but maybe moshi's developers could use my code to make moshi a multiplatform module for kotlin.

[sandwwraith]

kotlinx-serialization has okio integration since 1.4.0-RC: https://github.com/Kotlin/kotlinx.serialization/releases/tag/v1.4.0-RC

What exactly do you want from kotlinx-serialization side?

[fab1an]

It doesn't have the stream implementation using "startObject" etc I am looking for.

[hansenji]

I have a case where I have a small memory device and my json exceeds my memory space. Allowing me to decode/encode in chunks using tokens would allow me to actually use kotlinx-serialization. Ideally we want to support adapters like moshi and be kmp compatible.

See https://github.com/Kotlin/kotlinx.serialization/issues/2223

[TheKeeperOfPie]

I have a similar use case where I want to export a SQLite database to JSON, but I can't read the entire table into memory. For what it's worth, I think the APIs already exist to do what I want, as I can create a SerialDescriptor/KSerializer for my output format, which allows me to use beginStructure, and encode_ to write the values. I just can't create the output object directly in memory to pass to the current encode methods.

I considered faking this, by creating a marker class and then reading from the database directly in the KSerializer<Marker> subclass, as that gives me access to serialize(Encoder, ...) where I can just ignore the value passed and treat it like a streaming token API. But this seemed like a pretty severe mis-use of it.

All I would need is a way to get a StreamingJsonEncoder, which is currently internal API and too difficult to copy out into my own source.