ort
ort copied to clipboard
oss-review-toolkit
Potential SnakeYaml version upgrade issue
The latest release versions of SnakeYaml, the YML parser used by ORT, it is no longer possible to parse ORT files larger than 3 MB in size
The symptom is an exception similar to this: `com.fasterxml.jackson.databind.JsonMappingException: The incoming YAML document exceeds the limit: 3145728 code points.
Caused by: com.fasterxml.jackson.dataformat.yaml.JacksonYAMLParseException: The incoming YAML document exceeds the limit: 3145728 code points. at [Source: (BufferedInputStream); line: 83202, column: 17] at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:409) at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:308) at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:176) at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129) at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:313) ... 89 more Caused by: org.yaml.snakeyaml.error.YAMLException: The incoming YAML document exceeds the limit: 3145728 code points. at org.yaml.snakeyaml.scanner.ScannerImpl.fetchMoreTokens(ScannerImpl.java:335) at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:256) at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingValue.produce(ParserImpl.java:680) at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:171) at org.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:181) at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:403) ... 93 more
`
The problem seems to be introduced as part of the fix for CVE-2022-25857
There is currently no other workaround except fixing the snakeyaml version to 1.30
The latest release versions of SnakeYaml, the YML parser used by ORT
Minor clarification: ORT is not using SnakeYaml directly, but ORT is using Jackson which in turn uses SnakeYaml.
it is no longer possible to parse ORT files larger than 3 MB in size
That limit was introduced as part of SnakeYaml 1.32. However, I just verified that ORT's main branch uses only up to SnakeYaml 1.31 (via ./gradlew allDependencies | grep snakeyaml).
There is currently no other workaround except fixing the snakeyaml version to 1.30
This sentence confuses me now, because the 3MB limit has been introduced with version 1.32. So using version 1.31 (as we do) should be fine. Can you please clarify?
We will eventually want to upgrade anyway and I could not find a way to override the limit. Not being able to parse YAML files larger than 3MB would make the file format useless for us so I have asked the devs if they can confirm that my assumption is correct: https://github.com/FasterXML/jackson-dataformats-text/pull/335#issuecomment-1251631287
As explained here we could always use Gradle to force the transitive SnakeYaml version to 1.31 (or even 1.30), as most users probably aren't affected by the evoked CVEs anyway.
PS: Slightly off-topic, but with all the removals of the custom deserializers recently, I'm really looking forward to https://github.com/oss-review-toolkit/ort/issues/3904 which would solve also this issue for good.
Efforts to make the size limit configurable are now tracked in https://github.com/FasterXML/jackson-dataformats-text/issues/337.
PS: Slightly off-topic, but with all the removals of the custom deserializers recently, I'm really looking forward to #3904 which would solve also this issue for good.
Apart from potential technical issues we would first have to migrate all config files to use camel case, I think that's still a long way to go.
Efforts to make the size limit configurable are now tracked in FasterXML/jackson-dataformats-text#337.
This new feature is being made use of in https://github.com/oss-review-toolkit/ort/pull/6124.