MappingCouchbaseConverter.read() is not recursive

[JesusTheHun]

Using Cluster.query() I'm parsing the result into classes that have child object. A quick exemple to visualise :

@Data
public class A {
    String someProp;
    SubA nestedObject;

    @Data
    public static SubA() {
        String someNestedProp;
    }
}

Doing converter.read(A.class, document) leads to nestedObject being a HashMap<String, String> instead of a SubA as anyone would expect. Am I doing this wrong or are my expectations correct ?

[mikereiche]

I made an ARepsitory and did a save followed by a find.

git clone [email protected]:spring-projects/spring-data-couchbase.git

package org.springframework.data.couchbase.domain;

import lombok.AllArgsConstructor;
import lombok.Data;
import org.springframework.data.annotation.Id;
import org.springframework.data.couchbase.core.mapping.Document;

@Document
@Data
@AllArgsConstructor
public class A {
  @Id
  String someProp;
  SubA nestedObject;

  public String getId() {
    return someProp;
  }

  public String toString(){
    return "A{ someProp="+someProp+" nestedObject="+nestedObject+"}";
  }

  @Data
  @AllArgsConstructor
  public static class SubA {
    String someNestedProp;
    public String toString(){
      return "SubA{ someNestedProp="+someNestedProp+"}";
    }
  }

}

package org.springframework.data.couchbase.repository;

import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.couchbase.config.AbstractCouchbaseConfiguration;
import org.springframework.data.couchbase.domain.A;
import org.springframework.data.couchbase.domain.ARepository;
import org.springframework.data.couchbase.repository.config.EnableCouchbaseRepositories;
import org.springframework.data.couchbase.util.ClusterAwareIntegrationTests;
import org.springframework.data.couchbase.util.ClusterType;
import org.springframework.data.couchbase.util.IgnoreWhen;
import org.springframework.test.context.junit.jupiter.SpringJUnitConfig;

import java.lang.reflect.InvocationTargetException;
import java.util.Optional;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;

/**
 * Repository KV tests
 *
 * @author Michael Nitschinger
 * @author Michael Reiche
 */
@SpringJUnitConfig(CouchbaseARepositoryKeyValueIntegrationTests.Config.class)
@IgnoreWhen(clusterTypes = ClusterType.MOCKED)
public class CouchbaseARepositoryKeyValueIntegrationTests extends ClusterAwareIntegrationTests {

  @Autowired
  ARepository aRepository;

  @Test
  @IgnoreWhen(clusterTypes = ClusterType.MOCKED)
  void saveAndFindById() throws NoSuchMethodException, InvocationTargetException, IllegalAccessException {
    A.SubA subA = new A.SubA("this is someNestedProp");
    A a = new A("this is someProp", subA);
    a = aRepository.save(a);
    Optional<A> found = aRepository.findById(a.getId());
    assertTrue(found.isPresent());
    assertEquals(a,
        found.get());
    System.out.println("a="+a);
    System.out.println("a.suba="+a.getSomeProp());
    aRepository.delete(a);
  }


  @Configuration
  @EnableCouchbaseRepositories("org.springframework.data.couchbase")
  static class Config extends AbstractCouchbaseConfiguration {

    @Override
    public String getConnectionString() {
      return connectionString();
    }

    @Override
    public String getUserName() {
      return config().adminUsername();
    }

    @Override
    public String getPassword() {
      return config().adminPassword();
    }

    @Override
    public String getBucketName() {
      return bucketName();
    }

  }

}

src/test/resources/integration.properties

# Couchbase Integration-Test Default Properties
# If set to false, it is assumed that the host is managing the cluster and
# as a result no containers or anything will be spun up.
# Options: containerized, mocked, unmanaged
cluster.type=unmanaged <<<< change to unmanaged
# Default configs for both cases
cluster.adminUsername=Administrator
cluster.adminPassword=password
# Default configs for the mocked environment
cluster.mocked.numNodes=1
cluster.mocked.numReplicas=1
# Entry point configuration if not managed
# value of hostname and ns_server port
cluster.unmanaged.seed=10.144.220.101:8091 <<<< my server
cluster.unmanaged.numReplicas=0

I get the expected result:

a=A{ someProp=this is someProp nestedObject=SubA{ someNestedProp=this is someNestedProp}}
a.suba=this is someProp

From CouchbaseTemplate.support().decodeEntity()

T readEntity = converter.read(entityClass, (CouchbaseDocument) translationService.decode(source, converted));

[mikereiche]

the stacktrace from decodeEntity() will be this. See the two calls at read:243, MappingCouchbaseConverter? That's the recursion. The second call is initiated from read:Value:846 because the 'value' is a(nother) CouchbaseDocument.

readValue:845, MappingCouchbaseConverter (org.springframework.data.couchbase.core.convert) [2]
access$300:85, MappingCouchbaseConverter (org.springframework.data.couchbase.core.convert)
getPropertyValue:964, MappingCouchbaseConverter$CouchbasePropertyValueProvider (org.springframework.data.couchbase.core.convert)
getValueInternal:308, MappingCouchbaseConverter (org.springframework.data.couchbase.core.convert)
doWithPersistentProperty:276, MappingCouchbaseConverter$1 (org.springframework.data.couchbase.core.convert)
doWithPersistentProperty:268, MappingCouchbaseConverter$1 (org.springframework.data.couchbase.core.convert)
doWithProperties:360, BasicPersistentEntity (org.springframework.data.mapping.model)
read:268, MappingCouchbaseConverter (org.springframework.data.couchbase.core.convert)
read:243, MappingCouchbaseConverter (org.springframework.data.couchbase.core.convert)
readValue:846, MappingCouchbaseConverter (org.springframework.data.couchbase.core.convert) [1]
access$300:85, MappingCouchbaseConverter (org.springframework.data.couchbase.core.convert)
getPropertyValue:964, MappingCouchbaseConverter$CouchbasePropertyValueProvider (org.springframework.data.couchbase.core.convert)
getPropertyValue:913, MappingCouchbaseConverter$CouchbasePropertyValueProvider (org.springframework.data.couchbase.core.convert)
getParameterValue:74, PersistentEntityParameterValueProvider (org.springframework.data.mapping.model)
getParameterValue:53, SpELExpressionParameterValueProvider (org.springframework.data.mapping.model)
extractInvocationArguments:276, ClassGeneratingEntityInstantiator (org.springframework.data.mapping.model)
createInstance:248, ClassGeneratingEntityInstantiator$EntityInstantiatorAdapter (org.springframework.data.mapping.model)
createInstance:89, ClassGeneratingEntityInstantiator (org.springframework.data.mapping.model)
read:265, MappingCouchbaseConverter (org.springframework.data.couchbase.core.convert)
read:243, MappingCouchbaseConverter (org.springframework.data.couchbase.core.convert)
read:200, MappingCouchbaseConverter (org.springframework.data.couchbase.core.convert)
read:85, MappingCouchbaseConverter (org.springframework.data.couchbase.core.convert)
decodeEntity:106, CouchbaseTemplateSupport (org.springframework.data.couchbase.core)

btw - The CouchbaseTemplateSupport object and its decodeEntity() method are public, so you shouldn't have to dig into MappingCouchbaseConverter.read(). https://github.com/spring-projects/spring-data-couchbase/issues/1210

[mikereiche]

If you post your code, I'll run it in the debugger to see what's going on.

[JesusTheHun]

Here is the code :

Class definition

@Data
abstract public class BaseManageableEntity {
    @Id
    protected String id;

    @Field("dag_paths")
    protected List<DagPath> dagPaths = new ArrayList<>();

    abstract public ManageableType getManageableType();

    @Data
    @AllArgsConstructor
    @EqualsAndHashCode
    public static class DagPath {
        private LinkType pathType;
        private List<DagNode> path;

        @Data
        @EqualsAndHashCode
        public static class DagNode {
            private String id;
            private ManageableType type;

            private DagNode() {}

            public static DagNode of(BaseManageableEntity manageable) {
                if (manageable.getId() == null) throw new IllegalArgumentException();
                if (manageable.getManageableType() == null) throw new IllegalArgumentException();

                var node = new DagNode();
                node.id = manageable.getId();
                node.type = manageable.getManageableType();

                return node;
            }
        }
    }
}

@Document
@Data
@EqualsAndHashCode(callSuper = true)
public class Foo extends BaseManageableEntity {
    String title;
    public ManageableType getManageableType() {
        return ManageableType. ManageableTypeOne;
    }
}

public enum ManageableType {
    ManageableTypeOne, ManageableTypeTwo
}

public enum LinkType {
    LinkTypeOne, LinkTypeTwo
}

// Utility functions
public <T> T toDomainEntity(byte[] obj, Class<T> clazz) {
    JsonObject json = serializer.deserialize(JsonObject.class, obj);
    Class<?> targetClass = Class.forName(json.get("_class").toString());

    if (!clazz.isAssignableFrom(targetClass)) {
        throw new IllegalArgumentException(String.format("Trying to assign %s to %s", targetClass, clazz));
    }

    CouchbaseDocument document = new CouchbaseDocument()
            .setId(json.get(TemplateUtils.SELECT_ID).toString())
            .setContent(json);
    return (T) couchbaseTemplate.getConverter().read(targetClass, document);
}

public <T> List<T> toDomainEntities(List<byte[]> rows, Class<T> clazz) {
    return rows.stream().map(obj -> toDomainEntity(obj, clazz)).collect(Collectors.toList());
}

// Execution
QueryResult results = cluster.query(localQuery, QueryOptions.queryOptions().parameters(JsonObject.from(localMap)).adhoc(isAdHoc()));
return toDomainEntities(results.rowsAs(byte[].class), clazz);

[mikereiche]

Could you provide an input data? Thanks.

[mikereiche]

The difference could be that the fields are lists of sub-documents. But it should still work

[JesusTheHun]

I have edited the code in my previous comment to be more complete. Very basic stuff for data input:

var a = new Foo();
a.setId("a");

var b = new Foo();
b.setId("b");

var dagPath = new BaseManageableEntity.DagPath(LinkType.LinkTypeOne, List.of(BaseManageableEntity.DagPath.DagNode.of(a), BaseManageableEntity.DagPath.DagNode.of(b)));
b.getDagPaths().add(dagPath);

[mikereiche]

Foo doesn't implement getManageableType() from BaseManageableEntity.

[JesusTheHun]

I've updated the code.

[mikereiche]

couchbaseTemplate.insertById()/findById() (which uses CouchbaseTemplateSupport.decodeEntity) seems to work:

  @Test
  @IgnoreWhen(clusterTypes = ClusterType.MOCKED)
  void saveAndFindById2() {

    Foo a = new Foo();
    a.setId("a");

    var b = new Foo();
    b.setId("b");

    BaseManageableEntity.DagPath dagPath = new BaseManageableEntity.DagPath(LinkType.LinkTypeOne,
        List.of(BaseManageableEntity.DagPath.DagNode.of(a),
            BaseManageableEntity.DagPath.DagNode.of(b)));
    b.getDagPaths()
        .add(dagPath);

    try {
      couchbaseTemplate.removeById(Foo.class).one(b.getId());
    } catch(Exception e){
      //System.err.println(e);
    }
    couchbaseTemplate.insertById(Foo.class).one(b);

    var c = couchbaseTemplate.findById(Foo.class).one(b.getId());
    System.err.println("Foo: " +c);
    System.err.println("DagPath: "+c.getDagPaths());

    couchbaseTemplate.removeById(Foo.class).one(b.getId());
  }

Foo: Foo(title=null, mt=ManageableTypeOne)
DagPath: [BaseManageableEntity.DagPath(pathType=LinkTypeOne, path=[BaseManageableEntity.DagPath.DagNode(id=a, type=ManageableTypeOne), BaseManageableEntity.DagPath.DagNode(id=b, type=ManageableTypeOne)])]

Where does 'serializer' in toDomainEntity come from?

[JesusTheHun]

Where does 'serializer' in toDomainEntity come from?

It comes from cluster.environment().jsonSerializer().

[JesusTheHun]

I tried to use CouchbaseTemplateSupport but for that I needed to upgrade to 4.3.1 and then I met the breaking change exposed in #1315

[mikereiche]

The issue here (in 1312) is that in the CouchbaseDocument Foo, the List<> dagPath is an ArrayList (maybe it should be a CouchbaseList?) and it falls through to the else. I'll look into it.

In the meantime, can you use either the template or repository? Or maybe we can skip the CouchbaseDocument and go from use a String as the source.

	private <R> R readValue(Object value, TypeInformation<?> type, Object parent) {
		Class<?> rawType = type.getType();

		if (conversions.hasCustomReadTarget(value.getClass(), rawType)) {
			return (R) conversionService.convert(value, rawType);
		} else if (value instanceof CouchbaseDocument) {
			return (R) read(type, (CouchbaseDocument) value, parent);
		} else if (value instanceof CouchbaseList) {
			return (R) readCollection(type, (CouchbaseList) value, parent);
		} else {
			return (R) getPotentiallyConvertedSimpleRead(value, rawType);
		}
	}

[mikereiche]

Replace your toEntity() with this:

  public <T> T toDomainEntity(byte[] obj, Class<T> clazz) throws RuntimeException {
    JsonObject json = serializer.deserialize(JsonObject.class,
        obj);
    Class<?> targetClass = null;
    String id = null;
    long cas = 0;
    try {
      targetClass = Class.forName(json.get("_class")
          .toString());
      id = json.getString("__id");
      cas = json.getLong("__cas");
    } catch (ClassNotFoundException e) {
      throw new RuntimeException(e);
    }

    if (!clazz.isAssignableFrom(targetClass)) {
      throw new IllegalArgumentException(String.format("Trying to assign %s to %s",
          targetClass,
          clazz));
    }

    return (T) couchbaseTemplate.support()
        .decodeEntity(id,
            new String(obj),
            cas,
            targetClass);
 }

bme: Foo{title: null, mt: ManageableTypeOne, dagPaths: [DagPath{DagNodes : [DagNode{id : a}, DagNode{id : b}]}]}

[mikereiche]

This looks to be the same issue as #1276

[mikereiche]

The row needs to be decoded by the translationService. The translationService can be @Autowired from the config (this is what CouchbaseTemplateSupport.decodeEntity() does. The ArrayList is replaced by a CouchbaseList of CouchbaseDocuments, which MappingCouchbaseConverter converts to a List.

return  (T)couchbaseTemplate.getConverter()
    .read(targetClass,
        (CouchbaseDocument)translationService.decode(  new String (obj), document));

[JesusTheHun]

After moving to 4.3.2-SNAPSHOT and using to the translation service, this issue is resolved ! Thank you Michael !

[mikereiche]

I'm going to keep this open as someone else hit this and I need to update the documentation.

[JesusTheHun]

I'm going to keep this open as someone else hit this and I need to update the documentation.

Is the documentation open to PRs ?

[mikereiche]

Yes. They live in https://github.com/spring-projects/spring-data-couchbase/tree/main/src/main/asciidoc btw - there is a PR for querydsl at https://github.com/spring-projects/spring-data-couchbase/pull/1330 I don't seem to be able to add you as a reviewer.

[JesusTheHun]

btw - there is a PR for querydsl at #1330 I don't seem to be able to add you as a reviewer.

Having a sample project would help a lot to understand the architecture. Also a draft of documentation to get the gist of the current capabilities

[mikereiche]

There are samples in src/test/java/org/springframework/data/couchbase/repository/query/CouchbaseRepositoryQuerydslIntegrationTests.java They run with

1. changing src/test/resources/integration.properties cluster.type=unmanaged cluster.unmanaged.seed=10.144.220.101:8091
2. mvn integration-test -Dit.test=CouchbaseRepositoryQuerydslIntegrationTests

Bonus - to only create the bucket once and not delete it at the end of the tests.

src/test/java/org/springframework/data/couchbase/util/UnmanagedTestCluster.java 63 bucketname = "my_bucket"; // UUID.randomUUID().toString();