Added ability to buffered read huge strings in custom KSerializers

[fred01]

#1987 Added method which allow to perform large string handling (decode base64 binary for example) at user level

[fred01]

Example usage - https://github.com/fred01/TestStreams/blob/master/src/main/kotlin/Main.kt

[fred01]

Do I need update API dumps if jvmApiCheck is passed?

[sandwwraith]

If check task is successful, then API dumps won't change on updating

[sandwwraith]

But :kotlinx-serialization-core:jvmApiCheck is not successful in your case

[fred01]

You right, sorry, my bad. Fixed

[slavonnet]

I think much better use InputStream (decodeFromStream) + define custom serializer for BASE64 element

In serializer you can use https://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/binary/Base64InputStream.html (from apache or android core)

or

create temporary limited size ByteArrayBuffer / BufferedInputStream read bytes windowed/chunked by buffer size , decode and save to bytearray or temp file (outputStream). After finish create String from ByteArray or file.

Dont use ...From/ToString because its cause AllRead() and create large String Object, You was need 6g buffer * 3 times (for full read, for convert and for object). Then you read in StreamMode, you can read by bytes and reuse one 64k temp buffer

Also look https://github.com/Kotlin/kotlinx.serialization/pull/2101/files#diff-187a6f057732ac528be3c0acb7f4b5a83225c046d211361d4114eae76dee6b7b This request have some idea parts

[sandwwraith]

@fred01 :kotlinx-serialization-json:jvmApiCheck as well

[fred01]

I think much better use InputStream (decodeFromStream) + define custom serializer for BASE64 element

In serializer you can use https://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/binary/Base64InputStream.html (from apache or android core)

or

create temporary limited size ByteArrayBuffer / BufferedInputStream read bytes windowed/chunked by buffer size , decode and save to bytearray or temp file (outputStream). After finish create String from ByteArray or file.

Dont use ...From/ToString because its cause AllRead() and create large String Object, You was need 6g buffer * 3 times (for full read, for convert and for object). Then you read in StreamMode, you can read by bytes and reuse one 64k temp buffer

Also look https://github.com/Kotlin/kotlinx.serialization/pull/2101/files#diff-187a6f057732ac528be3c0acb7f4b5a83225c046d211361d4114eae76dee6b7b This request have some idea parts

1. Current approach is decode/encode algorithm-agnostic, it not force you to use Base64. It's still possible to use Apache's Base64InputStream at user level, just feed chunks to it.

2. I use toString() only for current buffer which has 16kb max length

[sviridov-alexey]

Also look https://github.com/Kotlin/kotlinx.serialization/pull/2101/files#diff-187a6f057732ac528be3c0acb7f4b5a83225c046d211361d4114eae76dee6b7b This request have some idea parts

Yes, I looked closely at this PR before starting this one. Get some inspiration from it.

[sandwwraith]

Although check failing tests on non-JVM platforms