My thoughts on JSON in Dart

[DanTup]

I don't know if I can add anything that isn't already described in the README or #6 but since this repo is soliciting feedback I thought I'd dump my thoughts here.

The company I work for evaluated Dart seriously a few years back and there were a bunch of things that really put us off, one of which was how difficult it seemed to do something really basic - deserialise some JSON into a class. If you consider that Dart advertises itself as "batteries included" and pitches itself as a language for building web apps it seems crazy that something as simple as JSON is so complicated (and seemingly for technical reasons? minification?).

All the existing solutions are, IMO, bad:

• Hand-rolling is tedious and error-prone
• Mirrors bloat code (apparently; I've never actually tried to use them, I've just seen complaints about bloated JS and bad performance)
• Transformers are clunky / going away and hard to debug
• Using maps loses all type safety (and one of the reasons for picking Dart over plain JS is for types!)

I'm really not sure a perfect solution can be achieved without some support from the language/platform, but there are some solutions that might be tolerable:

Fix mirrors/runtime reflection to not suck. I'm not sure why mirrors (or something similar) can't be a good solution; having classes marked with some attribute then emitting some metadata about the mappings of unminified-to-minified names doesn't seem like a huge overhead. I recently wrote some C# reflection code using Bridge.net and it transpiled to JavaScript and worked perfectly, and the metadata included was not all that bad and the performance was excellent.

Code-gen. The built_value library sounded pretty promising to me, but there were some frustrations when I tried to use it:

1. It required a lot of "noise" in my class definitions
2. It required everything to be immutable (this further added to the boilerplate and noise) - I am a fan of immutability but sometimes it's not needed and this seemed to really complicate things for trivial cases
3. Even using a "watch" script it only updates when I hit save in my IDE, so I'm frequently staring at a bunch of red squiggles even after I've fixed the issue just because the generated files are out of date

If you really don't want to built anything into the platform for serialisation then I think code-gen is the best option; but in order for that to work well I think the analyzer needs support to be able to generate code without the need to save files (it already knows the contents of all the buffers). I understand this is a change from how things like built_value currently work (watching file system changes or just reading the files in a one-off script) but without it I think the dev experience is going to suck!

Whatever the solution, it needs to not require third party libraries. JSON support in a web app is too fundamental for devs to have to worry about libraries being abandoned, not tested properly, making breaking changes on a whim, etc. etc. For something so basic we really need something tested/maintained/supported by the Dart team.

[matanlurey]

Thanks for the feedback - this is exactly the kind of stuff I want to see.

[xealot]

A+ to this.

We recently tried a Dart experiment to see if it would be beneficial compared to a plain Typescript environment. The pain of serializing/deserializing dozens of REST APIs killed it for us. Dart is no longer under consideration because of the ergonomics of JSON serialization.

[kevmoo]

@xealot see https://github.com/dart-lang/sdk/issues/30242

It's clear we need to do work here.

[xealot]

Very nice, sub'd to that issue and voted. I'll keep my eyes open for updates in this space.

[matanlurey]

Yup yup.

I did a ~1 day experiment myself, this type communicating with the Discord API.

Here is the result: https://github.com/matanlurey/din

I ended up being able to mostly auto-generate APIs even for semi-complex endpoints:

User authored:

@Resource(
  root: 'channels',
  docs: '$_resourceBase/channel#channels-resource',
)
abstract class ChannelsResource {
  factory ChannelsResource(RestClient client) = _$ChannelsResource;

  /// Get a channel by [id].
  ///
  /// Returns either a [Channel] object.
  @Endpoint(
    method: 'GET',
    path: const [#id],
    docs: '$_resourceBase/channel#get-channel',
  )
  Future<Channel> getChannel({
    @required String id,
  });

  /// Creates a message in [channelId] with text [content].
  ///
  /// Returns a created [Message] object.
  @Endpoint.asJson(
    method: 'POST',
    path: const [#channelId, 'messages'],
    docs: '$_resourceBase/channel#create-message',
  )
  Future<Message> createMessage({
    @required String channelId,
    @required String content,
  });
}

Generated:

// GENERATED CODE - DO NOT MODIFY BY HAND

part of 'channel.dart';

// **************************************************************************
// Generator: ResourceGenerator
// **************************************************************************

class _$ChannelsResource implements ChannelsResource {
  _$ChannelsResource(this._restClient);
  final RestClient _restClient;
  @override
  Future<Channel> getChannel({String id}) {
    return _restClient
        .request(
          url: 'channels/$id',
          method: 'GET',
        )
        .then((json) => new Channel.fromJson(json));
  }

  @override
  Future<Message> createMessage({String channelId, String content}) {
    return _restClient.request(
      url: 'channels/$channelId/messages',
      method: 'POST',
      json: {
        'content': content,
      },
    ).then((json) => new Message.fromJson(json));
  }
}

[kevmoo]

@DanTup have you looked at https://github.com/dart-lang/json_serializable/

We're still using build scripts – but the impact on code tries to be light. Mutability allowed, etc.

https://github.com/dart-lang/json_serializable/blob/master/example/example.dart

[kevmoo]

@xealot - feel free to email me at [github alias] @google.com – I'd love to understand your evaluation...

[kevmoo]

I played with this a bit here: https://github.com/kevmoo/dart_json_perf_experiments

Using json_serializable to create a Dart data model that support JSON serialization.

Then wrote a hack-ish thing to skip Dart's JSON utils entirely.

Clearly get about 2x+ improved deserialization. The downside: you're skipping any/all data validation.

Are folks seeing much worse than 2x? How much? Would folks rather have 2x faster and lose type safety?

Just curious...

[matanlurey]

If you're interacting with JSON you don't expect type safety anyway.

If you do, you're probably writing a schema validator. But most rely on the API documentation. This is sort of an area like --trust-type-annotations, where a well-tested/documented application is willing to skip some checks for performance improvements.

[DanTup]

@kevmoo

I'm less interested in the "over HTTP" spec than just general JSON support, since there could be many variations in what the server is returning and it might be difficult to support them all (though if you're able to run Dart on the server, then maybe this'd work out nice).

@DanTup have you looked at https://github.com/dart-lang/json_serializable/

I had not - it's newer than my complaints :) I had a quick scan, and if I'm understanding correctly, it's a bit like built_json but without all the immutability stuff? Looks like it might be just the thing I want! Though I guess it still has this issue from above:

Even using a "watch" script it only updates when I hit save in my IDE, so I'm frequently staring at a bunch of red squiggles even after I've fixed the issue just because the generated files are out of date

To fix this would presumably need analyzer support for the code-gen scripts, so that when there are in-memory changes to files in the IDE, the scripts should be run to also update the in-memory version of the generated files (I don't know if you plan to do this, but if you don't, you really should! :-))

Would folks rather have 2x faster and lose type safety?

I don't totally understand what you mean here by lose type safety? (but IMO, performance is not the issue, having strongly-typed data for communication is).

@matanlurey

If you're interacting with JSON you don't expect type safety anyway.

Not sure I agree here (if I understand correctly). The whole point (for me) is to have type safety. Sure, if my server returns json that doesn't match the client-side classes, things will go bad, but that's expected. But when I ensure my server and client are talking the same version of the classes, type safety is exactly what I want. I want to pretend that HTTP thing between my client server isn't just a big string, and return typed objects from the server and then consume them on the client as if they had been in the same process all the time.

[itsjoeconway]

@kevmoo @matanlurey

The important elements are:

1. Not having to double-declare keys/properties. Best case scenario is the key and property names match. Worst case scenario is an annotation is needed to link a property to its key name.
2. Validating types. Type information is already available by virtue of typing the property; that information ought to be used to verify that the expected type is received. An exception ought to be thrown during object mapping with the following info: the invalid key, the property's expected type, and the type of the JSON value.
3. Object graphs. If an object has a reference to some other decodable object, then decoding should cascade. This includes lists of objects.
4. Optionally being able to 'merge' an object. That is, instead of creating a brand-new object, one might read JSON into an existing object for which the input JSON takes precedence. This is useful for keeping a source of truth object graph locally and the most important component of local caching in non-web applications.

Speed is pretty low priority. A network call is already being made; a few milliseconds difference on object mapping has no discernible impact. A reasonable implementation won't add any more overhead than that. If there is a scenario where performance matters, it's up to the programmer to make decisions about manually decoding, moving to another thread, or other optimizations.

That being said, it's important that objects that are manually decoded can still work together with auto-decoded objects. In other words, manually decoded objects can simply 'override' their auto behavior and retain access to value decoding methods, including decoding other decodable objects.

I've built this kind of layer in Objective-C, Swift and Dart (with mirrors on the VM, not on the client). I'd suggest looking at how it is accomplished with mirrors in Aqueduct: https://github.com/stablekernel/aqueduct/blob/master/lib/src/db/managed/object.dart.

[kevmoo]

FYI - https://github.com/dart-lang/json_serializable/blob/experiment.custom_json_writer/README.md