Support loading files that exceed 3MiB

[vlsi]

Describe the bug

snakeyaml added a limit 3MiB limit: https://github.com/snakeyaml/snakeyaml/commit/72dfa9f1074abe2b8a6c8776bee4476b0aed02e3, and https://github.com/snakeyaml/snakeyaml/commit/440d98e2ebd586aafd9034fc74a2a47c656eb0ce

This probably needs adjustments: https://github.com/charleskorn/kaml/blob/6a29c79d8ee15c255a58b588c75545bc05150756/src/jvmMain/kotlin/com/charleskorn/kaml/YamlParser.kt#L33

It causes decodeFromStream to fail for documents exceeding 3145728 code points:

org.snakeyaml.engine.v2.exceptions.YamlEngineException: The incoming YAML document exceeds the limit: 3145728 code points.
    at app//org.snakeyaml.engine.v2.scanner.ScannerImpl.fetchMoreTokens(ScannerImpl.java:289)
    at app//org.snakeyaml.engine.v2.scanner.ScannerImpl.checkToken(ScannerImpl.java:192)
    at app//org.snakeyaml.engine.v2.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:711)
    at app//org.snakeyaml.engine.v2.parser.ParserImpl.lambda$produce$1(ParserImpl.java:232)
    at [email protected]/java.util.Optional.ifPresent(Optional.java:178)
    at app//org.snakeyaml.engine.v2.parser.ParserImpl.produce(ParserImpl.java:232)
    at app//org.snakeyaml.engine.v2.parser.ParserImpl.peekEvent(ParserImpl.java:206)
    at app//com.charleskorn.kaml.YamlParser$peekEvent$1.invoke(YamlParser.kt:53)
    at app//com.charleskorn.kaml.YamlParser$peekEvent$1.invoke(YamlParser.kt:53)
    at app//com.charleskorn.kaml.YamlParser.checkEvent(YamlParser.kt:65)
    at app//com.charleskorn.kaml.YamlParser.peekEvent(YamlParser.kt:53)
    at app//com.charleskorn.kaml.YamlNodeReader.readMapping(YamlNodeReader.kt:97)
    at app//com.charleskorn.kaml.YamlNodeReader.readFromEvent(YamlNodeReader.kt:63)
    at app//com.charleskorn.kaml.YamlNodeReader.readNodeAndAnchor(YamlNodeReader.kt:43)
    at app//com.charleskorn.kaml.YamlNodeReader.readMapping(YamlNodeReader.kt:112)
    at app//com.charleskorn.kaml.YamlNodeReader.readFromEvent(YamlNodeReader.kt:63)
    at app//com.charleskorn.kaml.YamlNodeReader.readNodeAndAnchor(YamlNodeReader.kt:43)
    at app//com.charleskorn.kaml.YamlNodeReader.readMapping(YamlNodeReader.kt:112)
    at app//com.charleskorn.kaml.YamlNodeReader.readFromEvent(YamlNodeReader.kt:63)
    at app//com.charleskorn.kaml.YamlNodeReader.readNodeAndAnchor(YamlNodeReader.kt:43)
    at app//com.charleskorn.kaml.YamlNodeReader.readNode(YamlNodeReader.kt:39)
    at app//com.charleskorn.kaml.YamlNodeReader.readSequence(YamlNodeReader.kt:88)
    at app//com.charleskorn.kaml.YamlNodeReader.readFromEvent(YamlNodeReader.kt:62)
    at app//com.charleskorn.kaml.YamlNodeReader.readNodeAndAnchor(YamlNodeReader.kt:43)
    at app//com.charleskorn.kaml.YamlNodeReader.readMapping(YamlNodeReader.kt:112)
    at app//com.charleskorn.kaml.YamlNodeReader.readFromEvent(YamlNodeReader.kt:63)
    at app//com.charleskorn.kaml.YamlNodeReader.readNodeAndAnchor(YamlNodeReader.kt:43)
    at app//com.charleskorn.kaml.YamlNodeReader.readNode(YamlNodeReader.kt:39)
    at app//com.charleskorn.kaml.YamlNodeReader.readSequence(YamlNodeReader.kt:88)
    at app//com.charleskorn.kaml.YamlNodeReader.readFromEvent(YamlNodeReader.kt:62)
    at app//com.charleskorn.kaml.YamlNodeReader.readNodeAndAnchor(YamlNodeReader.kt:43)
    at app//com.charleskorn.kaml.YamlNodeReader.readMapping(YamlNodeReader.kt:112)
    at app//com.charleskorn.kaml.YamlNodeReader.readFromEvent(YamlNodeReader.kt:63)
    at app//com.charleskorn.kaml.YamlNodeReader.readNodeAndAnchor(YamlNodeReader.kt:43)
    at app//com.charleskorn.kaml.YamlNodeReader.readNode(YamlNodeReader.kt:39)
    at app//com.charleskorn.kaml.YamlNodeReader.read(YamlNodeReader.kt:37)
    at app//com.charleskorn.kaml.Yaml.parseToYamlNodeFromReader(Yaml.kt:64)
    at app//com.charleskorn.kaml.Yaml.decodeFromReader(Yaml.kt:51)
    at app//com.charleskorn.kaml.Yaml.decodeFromStream(Yaml.kt:47)
    at app//com.charleskorn.kaml.Yaml.decodeFromStream$default(Yaml.kt:46)

Reproduction repo

No response

Steps to reproduce

Use decodeFromStream for document exceeding 3MiB

Expected behaviour

There should be a way to configure LoaderOptions

Actual behaviour

Parsing large YAML documents fails with YamlEngineException: The incoming YAML document exceeds the limit: 3145728 code points exception

Version information

com.charleskorn.kaml:kaml:{require 0.56.0; reject _} -> 0.56.0
\--- com.charleskorn.kaml:kaml-jvm:0.56.0
     +--- org.jetbrains.kotlinx:kotlinx-serialization-core:1.6.2
     |    \--- org.jetbrains.kotlinx:kotlinx-serialization-core-jvm:1.6.2
     |         +--- org.jetbrains.kotlin:kotlin-stdlib:1.9.21 -> 1.9.22 (*)
     |         +--- org.jetbrains.kotlinx:kotlinx-serialization-bom:1.6.2
     |         |    +--- org.jetbrains.kotlinx:kotlinx-serialization-core-jvm:1.6.2 (c)
     |         |    \--- org.jetbrains.kotlinx:kotlinx-serialization-core:1.6.2 (c)
     |         \--- org.jetbrains.kotlin:kotlin-stdlib-common:1.9.21 -> 1.9.22
     |              \--- org.jetbrains.kotlin:kotlin-stdlib:1.9.22 (*)
     +--- org.jetbrains.kotlin:kotlin-stdlib:1.9.21 -> 1.9.22 (*)
     \--- org.snakeyaml:snakeyaml-engine:2.7

Any other information

No response

[charleskorn]

Thanks for the issue @vlsi. I don't have time to implement this myself at the moment - would you be interested in submitting a PR to address this?

[sschuberth]

would you be interested in submitting a PR to address this?

I've just created https://github.com/charleskorn/kaml/pull/543.