Improve gradle support

[andrew]

Support ext.libraries array if possible: https://github.com/wiyarmir/TravisForAndroid/blob/f839343d11d76f002ca057e591f0737aff1bf38d/build.gradle#L73-L131

[StephanErb]

Another similar usage can be found in the Apache Aurora repository (https://github.com/StephanErb/aurora/blob/50f47ccc957381f101ecbffa91db0f6776fe19ad/build.gradle#L340-L384).

Only those dependencies are found whose version number does not come from a variable: https://dependencyci.com/github/StephanErb/aurora

[sschuberth]

In general, I'm wondering why to bother re-implementing parsing of build.gradle files and not leave that to Gradle itself, and just call e.g. gradle dependencies and parse its output. Is it a requirement for this project that getting the dependencies must work without the respective dependency manager installed?

[andrew]

@sschuberth Because build.gradle is just groovy code and running gradle dependencies would open up a whole host of security issues with our current infrastructure

[sschuberth]

Ok, so it basically is a requirement to not run any third-party executables. However, "is just groovy code" is exactly the problem: You could pretty much implement anything in Gradle build files, and name your variables / properties like you want. There's no standard for using a map named ext.libraries. That said, I believe you would always run short if you just parse the build.gradle files yourself instead of letting Gradle interpret them for you.

[andrew]

The plan is to eventually use gradle property from within a VM to ensure it can't do anything worse than crash but getting that infrastructure setup will take time

[sschuberth]

I other words, it does not make sense (yet) to improve handling of Gradle by filing a PR that executes Gradle and parses the output, right?

Just for reference, that basically is how LicenseFinder works, except that it also depends on the Gradle project to have the license-gradle-plugin applied.

[phatblat]

Simply parsing build.gradle is going to miss a wide variety of cases, especially when there are dependencySubstitution rules in a project.

[sschuberth]

Exactly. I've improved the wording in my previous comment as I meant "parsing of Gradle's output". But that's prone to changes in Gradle's output and thus likely to break. Which is why in our Open Source Review Toolikit we use the Gradle Tooling API to query dependencies programmatically (including and substitutions or version conflict resolutions).

[phatblat]

There's already a breaking change in the output of the dependencies task. The new implementation configuration doesn't include transitive dependencies like compile used to.

As of 1.1.1, ShellExec was still using compile:

./gradlew dependencies --configuration compile
> Task :dependencies

------------------------------------------------------------
Root project
------------------------------------------------------------

compile - Dependencies for source set 'main' (deprecated, use 'implementation ' instead).
+--- org.jetbrains.kotlin:kotlin-stdlib:1.2.10
|    \--- org.jetbrains:annotations:13.0
+--- org.apache.commons:commons-exec:1.3
\--- org.jetbrains.kotlin:kotlin-reflect:1.2.10
     \--- org.jetbrains.kotlin:kotlin-stdlib:1.2.10 (*)

Now it uses implementation:

./gradlew dependencies --configuration implementation
> Task :dependencies

------------------------------------------------------------
Root project - 🐚 Gradle plugin with a simpler Exec task.
------------------------------------------------------------

implementation - Implementation only dependencies for source set 'main'. (n)
+--- org.jetbrains.kotlin:kotlin-stdlib:1.2.31 (n)
+--- org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.2.31 (n)
+--- org.jetbrains.kotlin:kotlin-reflect:1.2.31 (n)
\--- org.apache.commons:commons-exec:1.3 (n)

[phatblat]

Just looked at your init.gradle @sschuberth and this is exactly what I was thinking this project needs! However, I believe the logic in there could be rolled up into a Gradle init script (.jar) plugin with it's own versioning. Then, it could be more easily reused by this project.

[phatblat]

Taking a step back, one way to get a quick win with improving Gradle support would be to parse the generated .pom file for java projects that deploy to a maven repo. The difficulty with this is that you'd either need to build the project locally and grab the .pom file out of the build/ dir, or download it from a maven repo where the project is released.

It's possible to use Gradle for building projects on other platforms (I'm using it for iOS & Android), but the vast majority of Gradle projects I imagine are deploying to a maven repo and thus generating one of these .pom files containing dependencies.

I do think that the initscript approach that oss-review-toolkit is the best way to catch all the cases as Gradle is very flexible.

Any thoughts @andrew or @BenJam? I'd love to help work on adding Gradle support.