[Issue]: Avro schemas with references

[CROprogrammer]

What happened?

Oirignal discussion: https://github.com/sngular/kloadgen/discussions/396

Im using Apicurio schema registry 2.2.3.Final

Hi,

I was trying to test if Avro schema with a reference to other schema can be loaded to KLoadGen from schema registry, but unfortunately it cannot.

I was using schema Student:

{
  "type": "record",
  "namespace": "com.example",
  "name": "Student",
  "fields": [
    {
      "name": "name",
      "type": "string"
    },
    {
      "name": "age",
      "type": "int",
      "default": -1
    }
  ]
}

and schema Address that has reference to schema Student:

{
  "type": "record",
  "namespace": "com.example",
  "name": "Address",
  "fields": [
    {
      "name": "name",
      "type": "string"
    },
    {
      "name": "student",
      "type": "com.example.Student"
    }
  ]
}

I kept getting error message: org.apache.avro.SchemaParseException: "com.example.Student" is not a defined name. The type of the "student" field must be a defined name or a {"type": ...} expression.

JMX testplan: KLoadGen Test Plan.zip

KloadGen Version

KLoadGen 5.6.5

Relevant log output

2023-07-07 10:44:36,165 ERROR o.a.j.JMeter: Uncaught exception in thread Thread[#39,AWT-EventQueue-0,6,main]
org.apache.avro.SchemaParseException: "com.example.Student" is not a defined name. The type of the "student" field must be a defined name or a {"type": ...} expression.
	at org.apache.avro.Schema.parse(Schema.java:1676) ~[kloadgen-5.6.5.jar:?]
	at org.apache.avro.Schema$Parser.parse(Schema.java:1433) ~[kloadgen-5.6.5.jar:?]
	at org.apache.avro.Schema$Parser.parse(Schema.java:1421) ~[kloadgen-5.6.5.jar:?]
	at io.apicurio.registry.serde.avro.AvroSchemaUtils.parse(AvroSchemaUtils.java:74) ~[kloadgen-5.6.5.jar:?]
	at io.apicurio.registry.serde.avro.AvroSchemaParser.parseSchema(AvroSchemaParser.java:59) ~[kloadgen-5.6.5.jar:?]
	at io.apicurio.registry.serde.avro.AvroSchemaParser.parseSchema(AvroSchemaParser.java:38) ~[kloadgen-5.6.5.jar:?]
	at com.sngular.kloadgen.schemaregistry.impl.ApicurioSchemaRegistry.setSchemaBySchemaType(ApicurioSchemaRegistry.java:110) ~[kloadgen-5.6.5.jar:?]
	at com.sngular.kloadgen.schemaregistry.impl.ApicurioSchemaRegistry.getSchemaBySubject(ApicurioSchemaRegistry.java:97) ~[kloadgen-5.6.5.jar:?]
	at com.sngular.kloadgen.util.JMeterHelper.getParsedSchema(JMeterHelper.java:30) ~[kloadgen-5.6.5.jar:?]
	at com.sngular.kloadgen.extractor.extractors.ExtractorFactory.flatPropertiesList(ExtractorFactory.java:68) ~[kloadgen-5.6.5.jar:?]
	at com.sngular.kloadgen.extractor.SchemaExtractor.flatPropertiesList(SchemaExtractor.java:33) ~[kloadgen-5.6.5.jar:?]
	at com.sngular.kloadgen.property.editor.SerialisedSubjectPropertyEditor.actionPerformed(SerialisedSubjectPropertyEditor.java:79) ~[kloadgen-5.6.5.jar:?]
	at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1972) ~[?:?]
	at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2313) ~[?:?]
	at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:405) ~[?:?]
	at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:262) ~[?:?]
	at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:279) ~[?:?]
	at com.github.weisj.darklaf.ui.button.DarkButtonListener.mouseReleased(DarkButtonListener.java:72) ~[darklaf-core-2.7.3.jar:2.7.3]
	at java.awt.Component.processMouseEvent(Component.java:6620) ~[?:?]
	at javax.swing.JComponent.processMouseEvent(JComponent.java:3398) ~[?:?]
	at java.awt.Component.processEvent(Component.java:6385) ~[?:?]
	at java.awt.Container.processEvent(Container.java:2266) ~[?:?]
	at java.awt.Component.dispatchEventImpl(Component.java:4995) ~[?:?]
	at java.awt.Container.dispatchEventImpl(Container.java:2324) ~[?:?]
	at java.awt.Component.dispatchEvent(Component.java:4827) ~[?:?]
	at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4948) ~[?:?]
	at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4575) ~[?:?]
	at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4516) ~[?:?]
	at java.awt.Container.dispatchEventImpl(Container.java:2310) ~[?:?]
	at java.awt.Window.dispatchEventImpl(Window.java:2780) ~[?:?]
	at java.awt.Component.dispatchEvent(Component.java:4827) ~[?:?]
	at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:775) ~[?:?]
	at java.awt.EventQueue$4.run(EventQueue.java:720) ~[?:?]
	at java.awt.EventQueue$4.run(EventQueue.java:714) ~[?:?]
	at java.security.AccessController.doPrivileged(AccessController.java:399) ~[?:?]
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86) ~[?:?]
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:97) ~[?:?]
	at java.awt.EventQueue$5.run(EventQueue.java:747) ~[?:?]
	at java.awt.EventQueue$5.run(EventQueue.java:745) ~[?:?]
	at java.security.AccessController.doPrivileged(AccessController.java:399) ~[?:?]
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86) ~[?:?]
	at java.awt.EventQueue.dispatchEvent(EventQueue.java:744) ~[?:?]
	at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203) ~[?:?]
	at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124) ~[?:?]
	at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113) ~[?:?]
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109) ~[?:?]
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101) ~[?:?]
	at java.awt.EventDispatchThread.run(EventDispatchThread.java:90) ~[?:?]

Have you added your JMX Testplan or a replica?

• [X] I have added a JMX testplan

Have you added your Schema or a replica?

• [X] I have added the Schema

[AdrianLagartera]

Thank you for collaborate again with us, we will work on it, but will take some time, so as we said in the discussion attached, we recommend you use protobuf for this purpose meanwhile you wait for this feature.

[CROprogrammer]

Hi, what's the current status of this issue?

[jemacineiras]

Hi @CROprogrammer,

we are looking into it. Sorry for the delay, holidays and so..

[jemacineiras]

Hi @CROprogrammer,

as far as we know, and concern, when we request the schema from the server that should be send with fully resolved, I am still not sure that we should do this task from kloadgen. Can you provide a working example about how you uploaded your schemas into ApiCurio?

Cheers

[CROprogrammer]

I've tried using Apicurio Maven plugin to register schema with reference to other schema, and it works fine. When I try to get http://localhost:8080/apis/registry/v2/ids/globalIds/{globalId}/references I indeed can confirm that my schema has reference to other schema.

My pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.0.6</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.example</groupId>
    <artifactId>avro2java_maven</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>avro2java_maven</name>
    <description>avro2java_maven</description>
    <properties>
        <java.version>17</java.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro</artifactId>
            <version>1.11.0</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
            <plugin>
                <groupId>org.apache.avro</groupId>
                <artifactId>avro-maven-plugin</artifactId>
                <version>1.11.0</version>
                <configuration>
                    <sourceDirectory>${project.build.directory}/src/main/avro/</sourceDirectory>
                    <outputDirectory>${project.build.directory}/src/main/generated-sources/java/</outputDirectory>
                    <stringType>String</stringType>
                    <enableDecimalLogicalType>true</enableDecimalLogicalType>
                    <fieldVisibility>PRIVATE</fieldVisibility>
                </configuration>
            </plugin>
            <plugin>
                <groupId>io.apicurio</groupId>
                <artifactId>apicurio-registry-maven-plugin</artifactId>
                <version>2.3.1.Final</version>
                    <configuration>
                        <registryUrl>http://localhost:8080/apis/registry/v2</registryUrl>
                        <artifacts>
                            <artifact>
                                <groupId>test-group</groupId>
                                <artifactId>Address</artifactId>
                                <version>1.0</version>
                                <type>AVRO</type>
                                <file>
                                    ${project.basedir}/src/main/resources/schemas/Address.avsc
                                </file>
                                <ifExists>RETURN_OR_UPDATE</ifExists>
                                <canonicalize>true</canonicalize>
                                <references>
                                    <reference>
                                        <name>com.example.Student</name>
                                        <groupId>test-group</groupId>
                                        <artifactId>Student</artifactId>
                                        <version>1.0</version>
                                        <type>AVRO</type>
                                        <file>
                                            ${project.basedir}/src/main/resources/schemas/Student.avsc
                                        </file>
                                        <ifExists>RETURN_OR_UPDATE</ifExists>
                                        <canonicalize>true</canonicalize>
                                    </reference>
                                </references>
                            </artifact>
                        </artifacts>
                        <output />
                    </configuration>
            </plugin>
        </plugins>
    </build>
</project>

Schemas I've used:

{
  "type": "record",
  "namespace": "com.example",
  "name": "Student",
  "fields": [
    { "name": "name", "type": "string" },
    { "name": "age", "type": "int", "default" : -1 }
  ]
}

{
  "type": "record",
  "namespace": "com.example",
  "name": "Address",
  "fields": [
    { "name": "name", "type": "string" },
    { "name": "student", "type": "com.example.Student"}
  ]
}

[jemacineiras]

Aye, what I´m trying to say, we delegate in apicurio client to download the schema, so it should came complete. I will try your pom and build a full path for testing.