swift-foundation icon indicating copy to clipboard operation
swift-foundation copied to clipboard

Published 20 hours ago • verified publisher icon apple

JSONEncoder can't encode what JSONDecoder decoded (they disagree on limits...)

Open weissi opened this issue 9 months ago • 4 comments

Consider JSONs like

{
  "nestedArray": [
    {
      "nestedArray": [
        {
          "nestedArray": [
            {
              "nestedArray": [
                {
                  "nestedArray": [
                    {
                      "nestedArray": []
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}

but deeper (for example bug1.json which has 255 levels). Now consider this program which decodes a thing and then reencodes it:

import Foundation

struct Thing: Codable {
    var nestedArray: [Thing]?
}

do {
    let data = try Data(contentsOf: URL(fileURLWithPath: CommandLine.arguments.dropFirst().first ?? "/tmp/file.json"))
    let decoded = try JSONDecoder().decode(Thing.self, from: data)
    do {
        let encoded = try JSONEncoder().encode(decoded)
        print("OK", encoded)
    } catch {
        print("encode ERROR", error)
    }
} catch {
    print("decode (or earlier) ERROR:", error)
}

You'd expect that JSONEncoder can encode everything that JSONDecoder decoded. But alas, here's what we see:

$ swiftc -O test.swift && ./test bug1.json
encode ERROR invalidValue(test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([test.Thing(nestedArray: Optional([]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))]))])), Swift.EncodingError.Context(codingPath: [], debugDescription: "Unable to encode the given top-level value to JSON.", underlyingError: Optional(Error Domain=NSCocoaErrorDomain Code=3840 "Too many nested arrays or dictionaries." UserInfo={NSDebugDescription=Too many nested arrays or dictionaries.})))

bug1.json

weissi avatar Feb 15 '25 09:02 weissi

👀 The max number of depth is hardcoded as 512 for both decoding and encoding.

💭 Not yet tested, but my guess is that the value of depth for decoding starts with 0 but the one for encoding starts with 1, because JSONScanner increments depth after checking it, on the other hand, JSONWriter increments depth before checking it.

https://github.com/swiftlang/swift-foundation/blob/8c87e9eaacef5187a9e1a9f732348c274522085c/Sources/FoundationEssentials/JSON/JSONWriter.swift#L31-L50

YOCKOW avatar Mar 28 '25 09:03 YOCKOW

@YOCKOW Does this need a fix? If so, can I look into opening a PR to address this?

jevonmao avatar Apr 06 '25 01:04 jevonmao

@YOCKOW Does this need a fix? If so, can I look into opening a PR to address this?

Sure, you can. I'm assigning you to this issue, thank you.

YOCKOW avatar Apr 07 '25 08:04 YOCKOW

Re-opening this issue as the original change was reverted due to https://github.com/swiftlang/swift-foundation/issues/1304

jmschonfeld avatar May 23 '25 16:05 jmschonfeld