sdk icon indicating copy to clipboard operation
sdk copied to clipboard

Provide a top-level function in `dart:convert` for decoding JSON from binary

Open osa1 opened this issue 1 year ago • 3 comments

Currently there's no discoverable way to decode JSON from binary data, and users have to convert the binary data to a string first, then decode the string, which is inefficient.

I actually thought this is not possible to do because it's so well hidden. Only the backend implementers know how to do it. You have to fuse a Utf8Decoder with a JsonDecoder:

(const Utf8Decoder()).fuse(const JsonDecoder()).convert(...)

However if you look at the source code, it looks as if this does it in two steps: https://github.com/dart-lang/sdk/blob/418858b7254f05e0ede90c9ef4feac52d5315ea8/sdk/lib/convert/converter.dart#L67

Which confuses even SDK devs: https://dart-review.googlesource.com/c/sdk/+/371080/comment/0e91626e_fa2b694f

The reason why this works is that backends can "patch" the fuse implementation to return a converter that does the conversion in one step:

  • VM: https://github.com/dart-lang/sdk/blob/e23e032406e2ab41641a5ea16a5f7020e06ff83b/sdk/lib/_internal/vm/lib/convert_patch.dart#L44-L49

  • Wasm: https://github.com/dart-lang/sdk/blob/e23e032406e2ab41641a5ea16a5f7020e06ff83b/sdk/lib/_internal/wasm/lib/convert_patch.dart#L32-L37

(dart2js doesn't patch it as there's no API to decode directly from a Uint8Array or similar in the browser)

There's no way for a user to discover this and use fuse instead of doing two conversions. E.g. https://dart-review.googlesource.com/c/sdk/+/371080.

The fact that this is possible is also not documented anywhere as far as I can see.

I think ideally we should have top-level function in dart:convert, similar to jsonDecode.

An easier fix might be just documenting (at the dart:convert top-level) that fusing decoders this way is possible and potentially more efficient, and should be preferred when decoding binary to JSON.

osa1 avatar Jun 13 '24 07:06 osa1

Adding an external const Codec<List<int>, Object> utf8Json; seems possible and reasonable. It'll have to fall back on being a fuse of utf8 and json where we don't have a more efficient version.

lrhn avatar Jun 13 '24 12:06 lrhn

We could also expose JsonUtf8Decoder to go along with JsonUtf8Encoder, then we have the bits for JsonUtf8Codec.

Right?

kevmoo avatar Jun 27 '24 23:06 kevmoo

Probably, yes. If the _JsonUtf8Decoder exists and is designed well enough to be public, we could expose it directly. Otherwise it might needs some tweaks.

lrhn avatar Jun 28 '24 13:06 lrhn