avro serialization of pojo with dynamic @JsonAnyGetter fields

[antonymayi]

Is it possible to serialize pojo with dynamic fields implemented using @JsonAnyGetter? This currently fails:

static class POJO {
    private final Map<String, Object> params = new HashMap<>();
    @JsonAnyGetter public Map<String, Object> any() {
        return this.params;
    }
    @JsonAnySetter public void set(String name, Object value) {
        this.params.put(name, value);
    }
    public Object get(String name) {
        return this.params.get(name);
    }
}

public void testAvro() throws IOException {
    ObjectMapper jsonMapper = new ObjectMapper(new JsonFactory());
    ObjectMapper avroMapper = new ObjectMapper(new AvroFactory());
    AvroSchemaGenerator gen = new AvroSchemaGenerator();
    avroMapper.acceptJsonFormatVisitor(POJO.class, gen);
    AvroSchema schema = gen.getGeneratedSchema();
    POJO orig = jsonMapper.readValue("{\"foo\": \"a\", \"bar\": 2, \"baz\": false}", POJO.class);
    byte binary[] = avroMapper.writer(schema).writeValueAsBytes(orig);
    POJO clone = avroMapper.reader(POJO.class).with(schema).readValue(binary);
}

That throws following:

Caused by: java.lang.IllegalStateException: No field named 'bar'
at com.fasterxml.jackson.dataformat.avro.ser.ObjectWriteContext._reportUnknownField(ObjectWriteContext.java:132)
at com.fasterxml.jackson.dataformat.avro.ser.ObjectWriteContext.writeFieldName(ObjectWriteContext.java:67)
at com.fasterxml.jackson.dataformat.avro.AvroGenerator.writeFieldName(AvroGenerator.java:285)
at com.fasterxml.jackson.databind.ser.std.StdKeySerializers$StringKeySerializer.serialize(StdKeySerializers.java:220)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:610)
at com.fasterxml.jackson.databind.ser.AnyGetterWriter.getAndSerialize(AnyGetterWriter.java:61)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:694)
... 29 more

[cowtowncoder]

Interesting. I assume this should work, as long as schema has such field so looks like a bug to me.

[cowtowncoder]

Actually, no. I should have read it more closely: you are generating Avro schema from POJO, and it has no real properties, just any setter/getter. As a result, Avro schema does not have any properties; and with that there is no way to encode contents.

This should, however, work if schema in use did have properties matching what "any getter" provides.

[antonymayi]

the point was that the AvroSchemaGenerator fails generating the schema for @JsonAnyGetter fields. I would expect this results in a avro Map field representing the any params. I assume had there been this map in the schema generated, then it would have been possible to serialize/deserialize its values?

[cowtowncoder]

@antonymayi Yes, if there was a Map it'd work: but these are not identical cases: "any properties" do unwrapping, whereas Map is a single named value.

I could see possible feature or something, to allow translating intent so perhaps it'd work like you suggest, but then there'd be question of what logical name to use for that Map.

So I am open to suggestions but just not quite sure how this could be made to work in a consistent and reliable way.

[baharclerode]

@cowtowncoder I would argue that if people want to do that, they should just have the map property on their POJO and serialize that without @JsonAnyGetter / @JsonAnySetter. This removes the burden upon Jackson of trying to generate or figure out a logical name for the map field, and puts them in complete control of the schema. You also have the problem of the map needing to have a statically known value type - Avro can't handle a Map<String,Object> generally speaking, so people will already have to define their POJOs with avro in mind.

I could see an argument for making @JsonAnySetter work correctly, i.e. I have an avro object with schema A, and deserializing it into a POJO with schema B and @JsonAnySetter-- Jackson would use @JsonAnySetter to handle fields that exist in A but not in B. (Doesn't work currently, but would take ~3 lines in the annotation introspector to make work)

@JsonAnyGetter however, cannot work with statically generated schemas, since the schema would have to have all the fields defined in it before looking at the map.

[cowtowncoder]

@baharclerode I concur -- interesting note on @JsonAnySetter, although that is sort of on path on diminishing returns I guess.

I may file a new issue for potential continuation, close this one, but will keep this one open for a bit longer.

[cowtowncoder]

@baharclerode come to think of it, wouldn't @JsonAnySetter already work, as-is? Since logical binding is done by avro codec... I guess there are some cases where it now drops "skippable" properties, and those could instead be "saved"?

[baharclerode]

@cowtowncoder It does, except the AvroAnnotationIntrospector doesn't look for the annotation, and JacksonAnnotationIntrospector isn't installed by default in an AvroMapper. So a developer can either manually set that up, or the "making @JsonAnySetter work correctly" I was referring to was adding it to the AvroAnnotationIntrospector so that it just worked out of the box with an AvroMapper.

I've not looked into how the skipping functionality might affect things.

[cowtowncoder]

@baharclerode Ohhh. Actually, I think this calls for one change: I think most of the time users would default to AnnotationIntrospectorPair where avro-introspector has precedence over jackson one. This is relatively easy to do by inserting introspector, instead of replacing. But to keep things fully configurable it probably makes sense to allow couple of options, so that for example:

1. Default option (not passing any AnnotationIntrospector would default to inserting avro one
2. Passing explicit AI would simply set that introspector, replacing default

Looks like this is handled via AvroModule so things need not be passed via constructor.

[baharclerode]

Yeah, I think that would make the most sense for behavior.