Java Bindings: Generate a utility method for conversion of DamlRecord into any of the templates within the codegen

[StephenOTT]

When working with TransactionTree.class you the CreatedEvent.class that provides the arguments property which is DamlRecord.class / Value.interface.

There is boilerplate code currently required on each project to generate a generic method that will convert from DamlRecord into the expected type.

Example: The DamlRecord provides the Identifier and based on this identifier you can look up which template to convert into and then use MyTemplate.fromValue((treeEvent as CreatedEvent).arguments)

What is the problem you want to solve?

This is boilerplate code that is created so typing can occur.

What is the solution you would propose?

provide a utility method that will do the switch/case check on each of the templates in the package to do the conversion into the proper class.

something like

   fun convertToTemplate(record: DamlRecord, identifier: Identifier): Template {
        return when (identifier){
            MyTemplate.TEMPLATE_ID -> MyTemplate.fromValue(record)
            MyOtherTemplate.TEMPLATE_ID -> MyOtherTemplate.fromValue(record)
...
            else -> throw RuntimeException("Unexpected Identifier")
        }
    }

or

   fun convertToTemplate(record: DamlRecord): Template {
        return when (record.recordId.orElseThrow()){
            Organization.TEMPLATE_ID -> Organization.fromValue(record)
            LegalEntity.TEMPLATE_ID -> LegalEntity.fromValue(record)
            else -> throw RuntimeException("...")
        }
    }

---

Would also be the same request for dealing with Choice Arguments in ExercisedEvent.class (This also brings up a partial problem that Choice classes do not have a common interface vs the Template classes have the Template interface to work with.)

[stefanobaghino-da]

The Java codegen has a -d switch that allows you to generate a decoder class. I ran

daml codegen java -o foo -d foo.bar.Decoder .daml/dist/create-daml-app-0.1.0.dar

on create-daml-app and the result is as follow:

package foo.bar;

import com.daml.ledger.javaapi.data.Contract;
import com.daml.ledger.javaapi.data.CreatedEvent;
import com.daml.ledger.javaapi.data.Identifier;
import java.lang.IllegalArgumentException;
import java.util.HashMap;
import java.util.Optional;
import java.util.function.Function;
import user.Alias;
import user.User;

public class Decoder {
  private static HashMap<Identifier, Function<CreatedEvent, Contract>> decoders;

  static {
    decoders = new HashMap<Identifier, Function<CreatedEvent, Contract>>();
    decoders.put(Alias.TEMPLATE_ID, Alias.Contract::fromCreatedEvent);
    decoders.put(User.TEMPLATE_ID, User.Contract::fromCreatedEvent);
  }

  public static Contract fromCreatedEvent(CreatedEvent event) throws IllegalArgumentException {
    Identifier templateId = event.getTemplateId();
    Function<CreatedEvent, Contract> decoderFunc = getDecoder(templateId).orElseThrow(() -> new IllegalArgumentException("No template found for identifier " + templateId));
    return decoderFunc.apply(event);
  }

  public static Optional<Function<CreatedEvent, Contract>> getDecoder(Identifier templateId) {
    return Optional.ofNullable(decoders.get(templateId));
  }
}

Does that work for you?

[StephenOTT]

Can the decoder be activated as part of daml start ?

[StephenOTT]

@stefanobaghino-da I ran the decoder gen on my code and it only generates decoders for Template classes.

Missing are ContractKeys(types and classes (sometimes a key resolves to a type like String, and other times to complex classes like Tuple or full blown data classes)), Choice classes and support for ExercisedEvent

some observations:

1. CreatedEvent implements from interfaces with fields so you can interact with a mix of CreatedEvent and ExercisedEvent. But if I get a list of Contract.class as per the "fromCreatedEvent" in the Decoder, I lose navigation of content because Contract interface is empty.

2. In the conversion utils i built that are similar goal as the Decoder class above, I re-created the Template classes as "Data classes" and applied a TemplateData interface class. This allows creating a contract interface such as: (ContractKey interface could potentially be merged into the Contract interface and apply a null when no key is being used. Still playing with the structures)

interface Contract<T: TemplateData> {
    var id: String
    var data: T
    var agreementText: String?
    var signatories: Set<String>
    var observers: Set<String>
    var templateId: TemplateIdentifier
}

interface ContractKey<K: Any> {
    // K cannot be abstract classes. Example: It cannot be Map<String, String>, but it can be HashMap<String, String>
    var key: K?
}

interface TemplateData {

}

The above is done so you can work with a mix of contracts regardless of the data within. In the decoder class because you return "Contract" interface, which is empty, we cannot do the following because Contract does not define any of the contract common properties:

myMixedListOfContracts.flatMap { c -> c.signatories }.toSet() // "Get a list of signatories for the list of contracts"

3. the Decoder's fromtCreatedEvent() returns a Contract, but what we really need is the DamlRecord -> Template:

example of what i created:

fun DamlRecord.toTypedTemplate(): Template {
    return when (this.recordId.orElseThrow()) {
        Organization.TEMPLATE_ID -> Organization.fromValue(this)
...
        else -> throw RuntimeException("No template conversion available for identifier: ${this.recordId.orElseThrow()}")
    }
}

3.1.

When you are interacting with a TransactionTree.class, the data structure is:

public class TransactionTree {
    private final String transactionId;
    private final String commandId;
    private final String workflowId;
    private final Instant effectiveAt;
    private final Map<String, TreeEvent> eventsById;
    private final List<String> rootEventIds;
    private final String offset;

...

Where TreeEvent is the CreatedEvent/ExercisedEvent. So you would want to keep this structure. The TreeEvent turns into a CreatedEvent or ExercisedEvent and this is where you need to convert the DamlRecord into a Template.class.

This is why i implemented the TemplateData interface and applied it into a DTO so i could work with TransactionTrees in a typed manner and move the data around for processing:

data class TransactionTreeDto(
    val transactionId: String,
    val commandId: String,
    val workflowId: String,
    val effectiveAt: Instant,
    val eventsById: Map<String, TreeEventDto>,
    val rootEventIds: List<String>,
    val offset: String
) {
    companion object {
        fun fromTransactionTree(transactionTree: TransactionTree): TransactionTreeDto {
...
}

// "TemplateIdentifier" was created to serialize identifiers as objects but then we also found that DamlRecords return Identifier.class for Choice classes.  So likely needs to be re-named.
interface TreeEventDto {
    val templateId: TemplateIdentifier
    val witnessParties: List<String>
}

data class CreatedEventDto(
    override val witnessParties: List<String>,
    val eventId: String,
    override val templateId: TemplateIdentifier,
    val contractId: String,
    val arguments: TemplateData,
    val agreementText: String,
    val contractKey: Any?,
    val signatories: Set<String>,
    val observers: Set<String>
) : TreeEventDto {

    companion object {
        fun fromDamlCreatedEvent(createdEvent: CreatedEvent): CreatedEventDto {
...
}

4. ExercisedEvent.class's ExercisedResult property is returning the raw "Value" class:

public class ExercisedEvent implements TreeEvent {
    private final List<String> witnessParties;
    private final String eventId;
    private final Identifier templateId;
    private final String contractId;
    private final String choice;
    private final Value choiceArgument;
    private final List<String> actingParties;
    private final boolean consuming;
    private final List<String> childEventIds;
    private final Value exerciseResult;
...

The exerciseResult is a Value class and this proved difficult to manage and de/serialize for use because of the wide variation of results that Value returns. it could be a String, an Object, a List, nested data, etc etc.

I had to create a "Value.class" wrapper and use it such as

//Example of hard coded ContractId scenario
// Value is a custom dto to wrap the Value so we can see the Type info.
Value(exercisedEvent.exerciseResult.toProto().sumCase.name, exercisedEvent.exerciseResult.asContractId().orElseThrow().value)

But it would be much cleaner if there was some better handling and understanding of what the potential for these values will be. it is one of the few places (only places?) where Value is returned in such as raw manner and no further type handling is provided.

public abstract class Value {
    public Value() {
    }

    public static Value fromProto(ValueOuterClass.Value value) {
        switch (value.getSumCase()) {
            case RECORD:
                return DamlRecord.fromProto(value.getRecord());
            case VARIANT:
                return Variant.fromProto(value.getVariant());
            case ENUM:
                return DamlEnum.fromProto(value.getEnum());
            case CONTRACT_ID:
                return new ContractId(value.getContractId());
            case LIST:
                return DamlList.fromProto(value.getList());
            case INT64:
                return new Int64(value.getInt64());
            case NUMERIC:
                return Numeric.fromProto(value.getNumeric());
            case TEXT:
                return new Text(value.getText());
            case TIMESTAMP:
                return new Timestamp(value.getTimestamp());
            case PARTY:
                return new Party(value.getParty());
            case BOOL:
                return new Bool(value.getBool());
            case UNIT:
                return Unit.getInstance();
            case DATE:
                return new Date(value.getDate());
            case OPTIONAL:
                return DamlOptional.fromProto(value.getOptional());
            case MAP:
                return DamlTextMap.fromProto(value.getMap());
            case GEN_MAP:
                return DamlGenMap.fromProto(value.getGenMap());
            case SUM_NOT_SET:
                throw new SumNotSetException(value);
            default:
                throw new UnknownValueException(value);
        }
    }
...

Example that Value can return a "Party.class" compared to Template.class that are generated for Template data use a regular String to hold a Party.

I am not saying one is better than the other. But Value becomes difficult to write flexible code with, especially with more wide-open objects such as Optional, Map, GenMap, List, Enum, Variant, and Record where each ~may be holding other nested objects within..

My current workaround has been having to build DTOs for every Value class and build wrappers and transformers around to get them into typed formats and make them serializable.

(not sure if it would be possible given how the daml exerciseResult works, but) if exercisedResult was a generated class based on the exercised choice logic, then the Value could be transformed into a business class for consumption. If not possible then i guess the re-use of DTOs to package exerciseResult in each app would be required?

[StephenOTT]

Another Note about Decoder: Should be a ~option to have a fall back decoding. Such as a "RawDamlRecord.class" or a specific exception for a unexpected decoding:

When decoding the arguments from DamlRecord to an expected type, there is a scenario where the ledger ~may pass an unexpected type that the client is not filtering on. So if the decoder cannot decode the record into a Template, Choice, Result Type, etc, then it would be nice to have a fallback wrapper.