Have an option to define the location for the Avro files

[x80486]

Prerequisites

• [X] Have you checked if the feature exists in the latest version of the plugin? (See releases)
• [X] Did you check to see if an issue has already been submitted?
• [X] Are you reporting to the correct repository? Requests for functionality not currently provided by the Avro Java library are better submitted to the Apache Avro project.
• [X] Did you perform a cursory search?

For more information, see the CONTRIBUTING guide.

Is your feature request related to a problem? Please describe.

As it stands, applying only com.commercehub.gradle.plugin.avro, there is no way to tell where to look for the Avro files. I think having that flexibility should be great because not all projects will be able to default the location to main/avro/ and using com.commercehub.gradle.plugin.avro-base and redefining a task just for this is a little bit overkill.

Describe the solution you'd like

We just need a sourceFiles setting for the avro block in order to define where to search for those.

Describe alternatives you've considered

Using the com.commercehub.gradle.plugin.avro-base plugin and define a generateSchema task.

[davidmc24]

Hi! Thanks for the feature request. I'm not opposed to exploring ways to make this aspect of the plugin more configurable, but would like to consider if there are other alternatives first.

Currently, the conventions plugin automatically generates source directory paths for each source set. In many projects, this means that it supports both src/main/avro and src/test/avro, but it dynamically expands if additional source sets are configured. I don't want to lose the ability for the plugin to automatically scale to new source sets, so any enhancements we make would need to take that into account.

If you only have a small number of source sets that you want to specify an alternate source location, you can currently configure that at the task level. Something like this would add an additional location to the existing task. Any source locations that don't exist are ignored, so you don't need to use src/main/avro, for example.

generateAvroProtocol {
  source("src/otherAvroLocation")
}
generateAvroJava {
  source("src/otherAvroLocation")
}

If this is adequate for your use case, I'd be happy to document it more clearly in the README.

What sort of directory structure do you want to use?

[x80486]

I think the defaults are perfect for a project that only (or mainly) generate these and then a JAR file that exports the Java types to be used later on different projects (I think that's a common case, or what I'm exposed more in my experience), but for a project that cannot have the Avro files in the source sets, but on a centralized location, I think the plugin should provide that functionality.

For instance, specifying a path outside the project, or in ${project.rootDir}/assets/, etc.

[davidmc24]

Have you tried the approach that I mentioned in my previous comment? Does it work for you?

What do you think that your envisioned feature would look like? How would you expect to configure it in your build file?

Basing it off how source directories are configured in the base Java plugin, perhaps the most consistent approach would be to add support for an avro property on sourceSets. Assuming we could get that to work, it would look something like this:

sourceSets {
    main {
         java {
            srcDirs = ['/some/external/directory']
         }
    }
}

[everest7]

Hi @davidmc24, I put the .avsc file in the folder specified in the build.gradle. But the build tool still only recognizes the avsc file under src/main/avro. Are those settings correct? I've uploaded my build.gradle file in txt. Thanks in advance.

generateAvroProtocol {
    source("src/resources/avro")
}
generateAvroJava {
    source("src/resources/avro")
}

build.gradle.txt

[davidmc24]

Not seeing a problem with your config at first glance (not at a computer at the moment). I should have time to dig into it tonight and see if I can reproduce the problem with your build file.

On Sun, Mar 15, 2020 at 3:11 PM chen.weis [email protected] wrote:

Hi @davidmc24 https://github.com/davidmc24, I put the .avsc file in the folder specified in the build.gradle. But the build tool still only recognizes the avsc file under src/main/avro. Are those settings correct? I've uploaded my build.gradle file in txt. Thanks in advance.

generateAvroProtocol { source("src/resources/avro") } generateAvroJava { source("src/resources/avro") }

build.gradle.txt https://github.com/davidmc24/gradle-avro-plugin/files/4334879/build.gradle.txt

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/davidmc24/gradle-avro-plugin/issues/98#issuecomment-599251632, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADNKUME73KTE7FGDSREPFTRHUR6LANCNFSM4J7M6P2A .

-- David M. Carr [email protected]

[everest7]

Not seeing a problem with your config at first glance (not at a computer at the moment). I should have time to dig into it tonight and see if I can reproduce the problem with your build file. On Sun, Mar 15, 2020 at 3:11 PM chen.weis @.***> wrote: Hi @davidmc24 https://github.com/davidmc24, I put the .avsc file in the folder specified in the build.gradle. But the build tool still only recognizes the avsc file under src/main/avro. Are those settings correct? I've uploaded my build.gradle file in txt. Thanks in advance. generateAvroProtocol { source("src/resources/avro") } generateAvroJava { source("src/resources/avro") } build.gradle.txt https://github.com/davidmc24/gradle-avro-plugin/files/4334879/build.gradle.txt — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#98 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADNKUME73KTE7FGDSREPFTRHUR6LANCNFSM4J7M6P2A . -- David M. Carr [email protected]

Thanks for replying. I also tried adding new sourceSet with the following code and put the avsc file under resources/avro/. Still doesn't work.

sourceSets {
    extra {
        java.srcDirs += "src/main/resources"
    }
}

[davidmc24]

@everest7 Based on your initial posting, I am assuming that:

1. You are interested in generating Avro Java classes based on Avro schema or protocol files in a directory other than the default (src/main/avro), such as src/resources/avro.
2. You have attempted to do so with the attached Gradle build file, and believe it isn't working.

Please correct me if either of those assumptions is wrong. I took your attached build file, and for me it appears to be working fine. Notes below.

• You didn't specify a JDK version, so I used Zulu 12.0.2. (Support for Java 13 would reuqire a version of the plugin I plan to release this week)
• You didn't specify a Gradle version, so I used Gradle 5.6.4 (Gradle 6.x support would require a version of the plugin I plan to release this week)
• You didn't specify the Avro files in use, so I use some sample ones. Specifically, I grabbed https://github.com/davidmc24/gradle-avro-plugin/blob/master/src/test/resources/com/commercehub/gradle/plugin/avro/mail.avpr and https://github.com/davidmc24/gradle-avro-plugin/blob/master/src/test/resources/com/commercehub/gradle/plugin/avro/enumField.avsc and put them in src/resources/avro/.
• I encountered an error build/generated-main-avro-java/org/apache/avro/test/Mail.java:25: error: package org.apache.avro.ipc does not exist void send(org.apache.avro.test.Message message, org.apache.avro.ipc.Callback<java.lang.String> callback) throws java.io.IOException;. This error was encountered during compileJava, and indicates that the source generation was working properly based on the Avro files in the custom directory, as it was failing to compile a generated Java file. I resolved the failure by adding a line compile "org.apache.avro:avro-ipc:1.8.2" to the dependencies block, as compiling Avro protocols requires avro-ipc.
• Version 0.9.1 of the plugin is two years out-of-date. The latest release is 0.18.0, and 0.19.0 will be out within the week. See https://github.com/davidmc24/gradle-avro-plugin
• The line java.srcDirs += "build/generated-main-avro-java/com/nordstrom/loyalty" shouldn't be needed. The avro plugin should automatically add any output directories of the generateAvroJava task to the main sourceSet's source directories. Even if you were manually configuring the Avro task with avro-base, you'd want to set it to build/generated-main-avro-java, rather than a package-specific directory.

[davidmc24]

@everest7 Full sample project attached gradle-avro-plugin-98.zip

[larsduelfer]

We need to include .avsc files from a library in the java source code generation. We have reusable avro schemas that are referenced from other avro schemas that are part of different libraries. So far, we had to copy the files in every lib where is was used in order to generate the java files. However, we want to get rid of these duplications. Anybody else having this issue and solved it already?

[philipp94831]

@larsduelfer I think the discussion in #128 can help you