swagger-codegen icon indicating copy to clipboard operation
swagger-codegen copied to clipboard
verified publisher icon swagger-api

[ANY LANG] The incoming YAML document exceeds the limit: 3145728 code points.

Open lorre851 opened this issue 3 years ago • 4 comments

Description

When executing the following command: java -jar modules/swagger-codegen-cli/target/swagger-codegen-cli.jar generate -i ../swagger.json -l html2 -o target

The following exception occurs, no mather which language (-l) is exported to:

io.swagger.v3.parser.util.DeserializationUtils$SnakeException: Exception safe-checking yaml content  (maxDepth 2000, maxYamlAliasesForCollections 2147483647)
	at io.swagger.v3.parser.util.DeserializationUtils$CustomSnakeYamlConstructor.getSingleData(DeserializationUtils.java:438)
	at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:477)
	at org.yaml.snakeyaml.Yaml.load(Yaml.java:406)
	at io.swagger.v3.parser.util.DeserializationUtils.readYamlTree(DeserializationUtils.java:211)
	at io.swagger.v3.parser.util.DeserializationUtils.deserializeIntoTree(DeserializationUtils.java:143)
	at io.swagger.v3.parser.OpenAPIV3Parser.readContents(OpenAPIV3Parser.java:165)
	at io.swagger.v3.parser.OpenAPIV3Parser.readContents(OpenAPIV3Parser.java:104)
	at io.swagger.v3.parser.converter.SwaggerConverter.readResult(SwaggerConverter.java:111)
	at io.swagger.v3.parser.converter.SwaggerConverter.readLocation(SwaggerConverter.java:85)
	at io.swagger.parser.OpenAPIParser.readLocation(OpenAPIParser.java:16)
	at io.swagger.codegen.v3.config.CodegenConfigurator.toClientOptInput(CodegenConfigurator.java:612)
	at io.swagger.codegen.v3.cli.cmd.Generate.run(Generate.java:386)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.yaml.snakeyaml.error.YAMLException: The incoming YAML document exceeds the limit: 3145728 code points.
	at org.yaml.snakeyaml.scanner.ScannerImpl.fetchMoreTokens(ScannerImpl.java:342)
	at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:263)
	at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingValue.produce(ParserImpl.java:694)
	at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:185)
	at org.yaml.snakeyaml.comments.CommentEventsCollector$1.peek(CommentEventsCollector.java:57)
	at org.yaml.snakeyaml.comments.CommentEventsCollector$1.peek(CommentEventsCollector.java:43)
	at org.yaml.snakeyaml.comments.CommentEventsCollector.collectEvents(CommentEventsCollector.java:136)
	at org.yaml.snakeyaml.comments.CommentEventsCollector.collectEvents(CommentEventsCollector.java:116)
	at org.yaml.snakeyaml.composer.Composer.composeScalarNode(Composer.java:239)
	at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:208)
	at org.yaml.snakeyaml.composer.Composer.composeKeyNode(Composer.java:347)
	at org.yaml.snakeyaml.composer.Composer.composeMappingChildren(Composer.java:332)
	at org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:311)
	at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:212)
	at org.yaml.snakeyaml.composer.Composer.composeValueNode(Composer.java:357)
	at org.yaml.snakeyaml.composer.Composer.composeMappingChildren(Composer.java:336)
	at org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:311)
	at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:212)
	at org.yaml.snakeyaml.composer.Composer.composeValueNode(Composer.java:357)
	at org.yaml.snakeyaml.composer.Composer.composeMappingChildren(Composer.java:336)
	at org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:311)
	at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:212)
	at org.yaml.snakeyaml.composer.Composer.composeValueNode(Composer.java:357)
	at org.yaml.snakeyaml.composer.Composer.composeMappingChildren(Composer.java:336)
	at org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:311)
	at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:212)
	at org.yaml.snakeyaml.composer.Composer.composeValueNode(Composer.java:357)
	at org.yaml.snakeyaml.composer.Composer.composeMappingChildren(Composer.java:336)
	at org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:311)
	at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:212)
	at org.yaml.snakeyaml.composer.Composer.getNode(Composer.java:134)
	at org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:160)
	at io.swagger.v3.parser.util.DeserializationUtils$CustomSnakeYamlConstructor.getSingleData(DeserializationUtils.java:415)
	... 12 common frames omitted
14:39:19.137 [Thread-0] ERROR i.s.v.p.util.DeserializationUtils - Error parsing content
com.fasterxml.jackson.dataformat.yaml.JacksonYAMLParseException: The incoming YAML document exceeds the limit: 3145728 code points.
 at [Source: (StringReader); line: 99391, column: 16]
	at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:425)
	at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer._deserializeContainerNoRecursion(JsonNodeDeserializer.java:539)
	at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:98)
	at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:23)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
	at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4772)
	at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:3124)
	at io.swagger.v3.parser.util.DeserializationUtils.readYamlTree(DeserializationUtils.java:232)
	at io.swagger.v3.parser.util.DeserializationUtils.deserializeIntoTree(DeserializationUtils.java:143)
	at io.swagger.v3.parser.OpenAPIV3Parser.readContents(OpenAPIV3Parser.java:165)
	at io.swagger.v3.parser.OpenAPIV3Parser.readContents(OpenAPIV3Parser.java:104)
	at io.swagger.v3.parser.converter.SwaggerConverter.readResult(SwaggerConverter.java:111)
	at io.swagger.v3.parser.converter.SwaggerConverter.readLocation(SwaggerConverter.java:85)
	at io.swagger.parser.OpenAPIParser.readLocation(OpenAPIParser.java:16)
	at io.swagger.codegen.v3.config.CodegenConfigurator.toClientOptInput(CodegenConfigurator.java:612)
	at io.swagger.codegen.v3.cli.cmd.Generate.run(Generate.java:386)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.yaml.snakeyaml.error.YAMLException: The incoming YAML document exceeds the limit: 3145728 code points.
	at org.yaml.snakeyaml.scanner.ScannerImpl.fetchMoreTokens(ScannerImpl.java:342)
	at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:263)
	at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingValue.produce(ParserImpl.java:694)
	at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:185)
	at org.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:195)
	at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:419)
	... 16 common frames omitted
14:39:19.311 [Thread-0] WARN  io.swagger.v3.parser.OpenAPIV3Parser - Exception while parsing:
com.fasterxml.jackson.dataformat.yaml.JacksonYAMLParseException: The incoming YAML document exceeds the limit: 3145728 code points.
 at [Source: (StringReader); line: 99391, column: 16]
	at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:425)
	at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer._deserializeContainerNoRecursion(JsonNodeDeserializer.java:539)
	at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:98)
	at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:23)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
	at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4772)
	at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:3124)
	at io.swagger.v3.parser.OpenAPIV3Parser.readContents(OpenAPIV3Parser.java:167)
	at io.swagger.v3.parser.OpenAPIV3Parser.readContents(OpenAPIV3Parser.java:104)
	at io.swagger.v3.parser.converter.SwaggerConverter.readResult(SwaggerConverter.java:111)
	at io.swagger.v3.parser.converter.SwaggerConverter.readLocation(SwaggerConverter.java:85)
	at io.swagger.parser.OpenAPIParser.readLocation(OpenAPIParser.java:16)
	at io.swagger.codegen.v3.config.CodegenConfigurator.toClientOptInput(CodegenConfigurator.java:612)
	at io.swagger.codegen.v3.cli.cmd.Generate.run(Generate.java:386)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.yaml.snakeyaml.error.YAMLException: The incoming YAML document exceeds the limit: 3145728 code points.
	at org.yaml.snakeyaml.scanner.ScannerImpl.fetchMoreTokens(ScannerImpl.java:342)
	at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:263)
	at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingValue.produce(ParserImpl.java:694)
	at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:185)
	at org.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:195)
	at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:419)
	... 14 common frames omitted
14:39:19.314 [Thread-0] INFO  i.s.c.v.i.CodegenIgnoreProcessor - No .swagger-codegen-ignore file found.
Exception in thread "Thread-0" java.lang.RuntimeException: missing OpenAPI input!
	at io.swagger.codegen.v3.DefaultGenerator.generate(DefaultGenerator.java:777)
	at io.swagger.codegen.v3.cli.cmd.Generate.run(Generate.java:388)
	at java.base/java.lang.Thread.run(Thread.java:833)

The swagger.json for our monolith of an application is 145 288 lines long.

Swagger-codegen version

3.0.36

Swagger declaration file content or url

I am not a liberty to share this .json file due to an NDA. The file has 145 288 lines and is about 4MB in size.

Command line used for generation

java -jar modules/swagger-codegen-cli/target/swagger-codegen-cli.jar generate -i ../swagger.json -l html2 -o target

openjdk 17.0.5 2022-10-18 Ubuntu 22.04.1 LTS x86_64

Steps to reproduce
  1. execute the command above with a large swagger.json file
  2. see the exception occur that's mentioned above
Related issues/PRs

N/A

Suggest a fix/enhancement

Increase upper limit or make configurable.

lorre851 avatar Nov 21 '22 13:11 lorre851

snakeyaml introduced this change to patch CVE-2022-25857. Given the use case for swagger-codegen there should be a better default or a configuration option to control this.

skwashp avatar Nov 23 '22 02:11 skwashp

it seems like it has been fixed.

something like

/usr/bin/java -DmaxYamlCodePoints=99999999 -jar /opt/swagger-codegen/swagger-codegen-cli-3.jar generate ...

works without errors

oprudkyi avatar Mar 30 '23 18:03 oprudkyi

it seems like it has been fixed.

something like

/usr/bin/java -DmaxYamlCodePoints=99999999 -jar /opt/swagger-codegen/swagger-codegen-cli-3.jar generate ...

works without errors java -DmaxYamlCodePoints=99999999 -cp cus-openapi-generator-1.0.0.jar:openapi-generator-cli.jar
org.openapitools.codegen.OpenAPIGenerator generate Exception safe-checking yaml content (maxDepth 2000, maxYamlAliasesForCollections 2147483647) it also error

nishuiaee avatar Aug 18 '23 03:08 nishuiaee