introduce wrap_string and wrap_number

[rjbs]

I'm not suggesting this PR be taken as is, although I think it's okay.

I need to test third party APIs and ensure that they are issuing JSON conforming to specifications. This means knowing whether they return 3 or "3". To do this, I added a wrap_string and wrap_number routine that do nothing new. I can then wrap them in other code, like this:

use strict;
use warnings;
use Data::Dumper;
use JSON::PP;

$Data::Dumper::Sortkeys = 1;

my $JSON = JSON::PP->new;

package JSON::string {
  use overload '""' => sub { ${ $_[0] } }, fallback => 1;
  sub new { my $x = $_[1]; bless \$x, $_[0]; }
  sub value { ${ $_[0] } }
}

package JSON::number {
  use overload '0+' => sub { ${ $_[0] } }, fallback => 1;
  sub new { my $x = $_[1]; bless \$x, $_[0]; }
  sub value { ${ $_[0] } }
}

{
  my $str = \&JSON::PP::wrap_string;
  my $num = \&JSON::PP::wrap_number;

  local *JSON::PP::wrap_string;
  local *JSON::PP::wrap_number;

  {
    no warnings 'redefine';
    *JSON::PP::wrap_string = sub {
      my ($s) = @_;
      JSON::string->new($str->($s));
    };

    no warnings 'redefine';
    *JSON::PP::wrap_number = sub {
      my ($n, $b) = @_;
      $n = $num->($n, $b);
      JSON::number->new($num->($n, $b));
    };
  }

  my $json = q<{"num":123, "str":"this is a string"}>;
  warn Dumper({
    input  => $json,
    output => $JSON->decode($json)
  });
}

my $json = q<{"num":123, "str":"this is a string"}>;
warn Dumper({
  input  => $json,
  output => $JSON->decode($json)
});

Do you think this is something we can move forward with?

[haarg]

It would be possible to check what the JSON type a value came from using the same mechanism that JSON::PP uses to ensure round tripping. That relies on the internal flags perl has set, so it is rather icky. But it is already basically part of the interface. It might make sense to expose an API to check how JSON::PP will encode a given value, which would give the same answer as what you want to check.

As for the approach used in this PR, I would think it would make more sense as filter_json_string and filter_json_number callbacks, similar to filter_json_object and filter_json_single_key_object.

[rjbs]

Checking the scalar type is quite fragile. I'd have to do it immediately after decode and walk the returned structure to mark things in a robust way (like this). I can do that, of course, but this seemed like something that might be useful to provide generically in JSON::PP.

[rjbs]

Last night as I drifted off to sleep I realized that this would be much better done with something like filter_json_object, but I didn't know that this option even existed.

Let me know whether you'd rather have a patch that uses something like that, or whether I should just write my own, not-in-JSON::PP, wrapper around decode that immediately replaces string/number nodes, and I'll move ahead that way.

[haarg]

I agree with your concerns about checking the scalar type, but given that it's already used, exposing it as a set of functions doesn't seem unreasonable. And for your use, yeah it would involve walking the structure immediately after decoding.

I don't object to the approach similar to filter_json_object though.

[rjbs]

I've written this for use in the meantime: https://github.com/rjbs/JSON-Typist/blob/master/lib/JSON/Typist.pm

I will gladly redo this PR however Makamaka would like, if it's got hope of going into JSON::PP itself.

[charsbar]

Hi. I suppose the following code does (almost) the same as what you proposed, which works not only with JSON::PP but also (Cpanel::)?JSON::XS, without adding extra callbacks nor restriction for JSON::Typist that needs to run immediately after decoding.

A shortcut for this might be worth to add, maybe to JSON.pm, rather than to JSON::PP, but I suppose we don't need wrap_string and wrap_number. What do you think?

use JSON::PP; # (or (Cpanel::)?JSON::XS)
use JSON::Typist; # just for overloading

my $json = JSON::PP->new->filter_json_object(sub {
  my $obj = shift;
  no warnings 'numeric';
  for my $key (keys %$obj) {
    my $value = $obj->{$key};
    if (!ref $value) {
      if (length((my $dummy = "") & $value) # HAARG++
          && 0 + $value eq $value
          && $value * 0 == 0
      ) {
        $obj->{$key} = bless \$value, 'JSON::Typist::Number';
      } else {
        $obj->{$key} = bless \$value, 'JSON::Typist::String';
      }
    }
  }
  $obj;
});

my $data = $json->decode('{"foo":123, "bar":"str", "baz": {"quuz": "123"}}');

say $json->convert_blessed->encode($data);

[charsbar]

Oops. No. It only works with objects = hashes. Forget about the above comment.

[pali]

Hi! Another way to ensure of generating correct json according to some type specification is to extend encode_json function to process second parameter which get type specification. And JSON::PP itself will convert input structure/value to correct type. I have some prototype for this written... What do you think? Example:

    encode_json(
        [ { key1 => '11', key2 => [ 12, 13 ] } ],
        [ { key1 => 'INT', key2 => [ 'STRING', 'INT' ] } ]
    );

will return:

'[{"key2":["12",13],"key1":11}]'

[charsbar]

@pali, I don't think it's a good idea. It's confusing and likely to break interoperability.

[pali]

It is for more complicated code... E.g. when you get perl structure from another functions or other modules and want to generate correct JSON.

Or when you got perl scalar from another function and you need to ensure that it will be int in json (not string)... and you do not know what function returns (either stringified number or number)...

I really do not know any other way how to generate correct json (with correct types) from perl structure which comes from other functions other modules.

In perl you do not distinguish between string and int, so it is not possible to write JSON serializer in Perl correctly if you do not provide information if type is string or int.

[charsbar]

@pali, you might want to look at JSON::Schema::Fit (and thus JSON::Schema). I'm not particularly a big fan of those modules, but they are more standard (or at least comply with some standard), and don't break existing codes. (I understand your concerns, which are quite common.)

[pali]

@charsbar: Already looked at it and also other cpan modules. JSON::Schema is just for finite/non-recursive structures, but perl structures can be DAG... Anyway, my proposal is backward compatible (adds optional second argument for json_encode, when not present code behave as before)... I will post code as new pull request (but it is not mean to merge, just proposal/example), so we can discuss about it better...

[pali]

Merge request created: https://github.com/makamaka/JSON-PP/pull/32 Move discussion here.