Serialization vs. deserialization of Optional types

[dusik66]

Optional types has not compatible serialization and deserialization.

Deserialization:

• missing field - java field is null / unchanged,
• "null" - Optional.empty(),
• "any value" - deserialized to java field.

Serialization:

• null field - is not serialized / missing in json (except nilable),
• Optional.empty() - is not serialized / missing in json (except nilable),
• Optional.of("any value") - is serialized to "any value".

Problem is deserialization of null value. It is deserialized into Optional.empty(). But there is no way serialize it back to null value.

I propose serialize Optional.empty() into null value. This behavior is important for implementation of JSON Merge Patch - see https://tools.ietf.org/html/rfc7386 .

[rmannibucau]

Hi @dusik66 , this should actually be the case if you enable to serialize null values (https://github.com/eclipse-ee4j/jsonb-api/blob/master/api/src/main/java/jakarta/json/bind/JsonbConfig.java#L219). Default does not as expected I think.

[dusik66]

If you use JsonbConfig.withNullValues() then both null field and Optional.empty() are serialized to same json value - null. And it is still not correct. You have to distinguish between them. Deserialization do, but serialization don't.

[rmannibucau]

@dusik66 you can always use a custom serializer to do it but I have to admit I don't see an use case for it. Optional are meaningless in JSON in all cases, they are just a wrapper to test programmatically null or not so it must behave exactly as null. Deserialization only default to empty() by contract but there is no optional wrapper in JSON so it behaves as null. Do you have an example where JSON Merge Patch fails with it? Do you mean Optional gets null instead of empty()? if so it is a bug on JsonPatch (or lack of specification) but it should just be aligned on the other parts of JSON-B IMHO.

[dusik66]

I will try to describe where the JsonB fails.

There are two systems communicating via rest and using patch merge specification to modifying entities. There is a entity with 3 fields - title, firstname and lastname. First system wants to call second system with a message to delete title, set new value for firstname and leave unchanged lastname. Based on the patch merge specification the json of the message looks like this:

{
  "title": null,
  "firstname": "Tom"
}

Systems are using following data class to serialize the message:

public class Patch {
  Optional<String> title;
  Optional<String> firstname;
  Optional<String> lastname;
}

Deserialization of the json message works fine. There will be object filled like this:

public class Patch {
  Optional<String> title;          = Optional.empty()
  Optional<String> firstname;      = Optional.of("Tom")
  Optional<String> lastname;       = null
}

Called system knows that the title should be cleared in the database (because of Optional.empty()), value of firstname should be changed to "Tom" (because of Optional.of("Tom")) and lastname leave unchanged (because of null).

Calling system has trouble, because there is no way to serialize the object via JsonB. Serialization is able to create:

{
  "title": null,
  "firstname": "Tom",
  "lastname": null
}

or

{
  "firstname": "Tom"
}

Unfortunately both are wrong.

[rmannibucau]

@dusik66 I think I see. Let me put some thoughts here to try to refine where the actual issue is:

1. you assume Optional.empty() and null are different for JSON, it is sadly not the case since both are equivalent to JsonValue.NULL_VALUE cause JSON does not have Optional. Note that it is the same in Java, Optional.empty() is a representation of null value. This means that your modelisation is not matching what you are expecting.
2. You are not using JSON-P JsonPatch here it seems (until I misunderstood something) so your diff is a custom modelisation which has the design bug of 1.
3. If you use setters and not only fields, the setX() call can be captured and you can flag the field as being set, meaning the null value associated to the flag "set" will be your "empty()" and a null value with the flag not set (false) will be your "null".

So overall - and once again, if I got it right - I think the issue is not in JSON-B really but more in your patching system which seems to rely on ternary value (null, empty() anything else) which is not properly modelised with JSON-B.

Does it make sense?

[dusik66]

ternary value (null, empty() anything else) which is not properly modelised with JSON-B

This is the point. JSON-B does not support ternary values. But JSON does - has value, is null, is missing. JSON-B deserialization supports it. But serialization don't, which is not correct, I think.

Beside this example. Serialization and deserialization should be inverse functions. In case of JSON-B they are not. Deserialization has a exception in the specification, but serialization don't.

The exception is java.util.Optional, OptionalInt, OptionalLong, OptionalDouble instances. In this case the value of the field is set to an empty optional value.

[rmannibucau]

JSON-B does not support ternary values

Well, strictly speaking it is not JSON-B but Java since object are static, JSON-B just respects that.

Serialization and deserialization should be inverse functions.

What would it mean? In java, empty() literally means "is null", it does not mean "is not there". Best you can do is to add a custom JsonbSerializer and a marker on your field saying @IgnoreWhenMissing and if so you don't write anything, but I'm not sure it is mainstream - actually it is against all the cases I saw using optional since 1.0 so i'm more convinced it is a particular case needing a custom serializer.

Using JSON-P patch format can be worth it too since it enables a more acucrate patch format since you associate to the pointer an operation so you don't assume the value is enough to determine the operation to do which is your issue today.

[dusik66]

Function D is inverse to function S only if applying function S on result of application function D will have same result as the original.

m=S(D(m))

• Lets start with json message m.
• Then deserialize it into java object.
• Then serialize java object to json let's say n.
• Both json messages m and n have to be equal.

Java serialization (classes ObjectInputStream and ObjectOutputStream) comply this paradigm. Unfortunately JSON-B don't. This is very important feature for serialization. You can rely that after deserialization and serialization you will receive back original data. Without this the whole java serialization concept would fails.

Let's correct this bug by adding one sentence to the specification 3.14.1.:

The result of serializing a java field with a null value is the absence of the property in the resulting JSON document. The exception is java.util.Optional, OptionalInt, OptionalLong, OptionalDouble instances. In this case the empty optional value is set as null value in a JSON document.

[rmannibucau]

@dusik66 how does java serialization handle Optional values? ;)

That said, your proposal breaks most apps using Optional, as already mentionned, it is your usage but not the intended usage of optional by java. I recommend you to read this thread http://mail.openjdk.java.net/pipermail/lambda-dev/2012-September/005946.html (not this particular message but the whole thread).

Concretely, you merge the concept of java optionality and JSON optionality but you probably noted they are generally different. Take the simple case where you use JSON to modelize domain objects, then you want optional to be part of your domain API (just java side) not JSON API, your proposal breaks that. And once again, there is no blocker to implement what you want with JSON-P + JSON-B, all is there for your particular usage. So if we want to handle your case at JSON-B level, we must introduce a new API, like a new property to handle nullValues configuration dynamically per field value - but note it is exactly what (de)serializers are - or we would induce unintended regressions.

[dusik66]

It is not necessary define new API per field. Just new global property in JsonbConfig similar to withNullValues().

Patch merge specification is just example of the JSON usage.

In general I think that serialization of deserialized message should be equal as original one - m=S(D(m)). If not, it is not correct.

[rmannibucau]

@dusik66 both or none are needed cause it is how everything is designed, works and as mentionned it would break apps. Optional should really respect nullValues setup by default cause Optional is to avoid NPE by design, not to materialize null in json. Once again JSON merge spec is already well supported by jsonb, either using setters or (de)serializers but not as you tried which really looks a counter usage more I review optional usages in apps. Can you give a try to fix your modelling?

[dusik66]

not to materialize null in json.

I don't understand why deserialization is converting null in json to Optional.empty()? Why not simply to java null?

[rmannibucau]

@dusik66 cause Optional design is to avoid any null value (idea being, if you never use null you can't get any NPE in your app). So null for Optional field/vars is considered as a bug.