Add support for NDJSON: newline-delimited JSON

[petdance]

I'm using Solr and their update API uses NDJSON, or newline-delimited JSON, for its streaming updates. It would help me if Mojo::UserAgent supported it. I'm glad to write code/docs/tests for it, if it's something that the Mojolicious project is interested in.

In short, NDJSON has one JSON document per line, followed by a newline. Instead of sending this Perl struct:

[
    { foo => 1 },
    { bar => { this => 10 } }
]

as

[{"foo":1},{"bar":{"this":10}}]

NDJSON would send it as:

{"foo":1}
{"bar":{"this":10}}]

The goal would be that I could post updates to Solr using:

my $res = $ua->post( $update_url => ndjson => $commands );

In the meantime, I'm using:

my $ndjson = join( "\n", map { encode_json($_) } @{$commands} );
my $res = $ua->post( $update_url => { 'Content-Type' => 'application/json' } => $ndjson );

The code to to this in Mojo::UserAgent looks to be pretty trivial, roughly:

sub _ndjson {
  my ($self, $tx, $data) = @_;
  my $ndjson_data =
    ref($data) eq 'ARRAY'
      ? join("\n", map { encode_json $_ } @{$data}
      : encode_json $data;
  _type($tx->req->body($ndjson_data)->headers, 'application/json');
  return $tx;
}

Of course, the real work is updating docs and tests.

I'd be glad to do the work and create the PR if this is something the Mojo project is interested in.

[shadowcat-mst]

This seems like something you could prototype as a Role::Tiny based Mojo::UserAgent::Role::NDJSON - and then apply to your $ua with

$ua->with_roles('+NDJSON');

which strikes me as a pretty neat thing to have, btw, and whether it eventually ends up in core or not, having the rule would be a great proving ground and a way to let people get access to it who don't want to upgrade Mojolicious just yet.

(I love NDJSON as a format for a lot of things, btw, I certainly wouldn't personally mind it in core though I don't get a say on it, but either way, "role first" strikes me as the best way forwards)

[Grinnz]

Custom generators can be easily added by third party modules:

$ua->transactor->add_generator(ndjson => sub ($t, $tx, $data) {
  # the code you wrote above
});

[Grinnz]

Found the guide: https://metacpan.org/pod/Mojolicious::Guides::Cookbook#Content-generators

[Grinnz]

On the feature request, it might be reasonable to add in core if and when there's suitable demand for it, but in the meantime it is easy to prototype via CPAN using whatever interface you find convenient.

[petdance]

Thanks for the pointers.

Adding the generator seems like the way to go, but I'd hate to have to add the ->add_generator everywhere. My first notion would be to make my own subclass My::Mojo::UserAgent that does the ->add_generator call in the overloaded constructor. Is there a more Mojo way I should do instead?

[kraih]

Is there a more Mojo way I should do instead?

Roles.

[Grinnz]

A role would allow it to be composed in conjunction with other Mojo::UserAgent roles hence preferring it over subclassing. It means you have to add a method to install the generator because roles can be composed before or after construction, though.

Another option is exporting a method to install the generator, to be used on any Mojo::UserAgent object. https://metacpan.org/pod/Mojo::IOLoop::Subprocess::Sereal provides both options because I found the exported method simpler in the end for that use case, though it may be "uglier".