Recursive mapping issue (despite custom serializers)

[sdaschner]

Describe the bug Class structures that contain a recursive reference can't be mapped to JSON, even though when one defines custom JSON-B serializers or type adapters. This doesn't really make sense, since serializing (back) references to, for example, scalar values stops the recursion.

To Reproduce

Reproducer here: https://github.com/sdaschner/quarkus-playground/releases/tag/jsonb-recursive-mapping

mvn quarkus:dev
curl localhost:8080/json-test

(or run JsonIT in the project)

This will cause an exception like:

Caused by: jakarta.json.bind.JsonbException: Unable to serialize property 'schedule' from com.sebastian_daschner.coffee.JsonTestResource.Item
	at org.eclipse.yasson.internal.serializer.ObjectSerializer.lambda$serialize$0(ObjectSerializer.java:43)
[...]
	at org.eclipse.yasson.internal.serializer.ObjectSerializer.lambda$serialize$0(ObjectSerializer.java:41)
	... 25 more
Caused by: jakarta.json.bind.JsonbException: Recursive reference has been found in class class com.sebastian_daschner.coffee.JsonTestResource$Schedule.
	at org.eclipse.yasson.internal.serializer.RecursionChecker.serialize(RecursionChecker.java:36)
[...]

As you can see in the code (Schedule -> Item), the back reference Item#schedule is annotated with @JsonbTypeSerializer(ScheduleNameSerializer.class). Only if also the reference Schedule#item is annotated with a similar serializer, it works. Adding @JsonbTransient on the back reference also works.

Expected behavior

The following output would be expected from that test:

{
  "item": {
    "name": "Item",
    "schedule": "Name"
  },
  "name": "Name"
}

System information:

• Yasson Version: 3.0.3

Additional context Originally submitted in https://github.com/quarkusio/quarkus/issues/42442