Copy dependencies' jar files to where Eclipse expects them

[liblit]

I thought I'd take a swing at #742, in spite of my lack of knowledge about Eclipse PDE, build.properties files, etc. In a pristine WALA tree, importing into Eclipse leaves me with 136 plug-in errors. The first four of these are:

Description	Resource	Path	Location	Type
'source.target/dependency/com.ibm.wala.cast.jar' build entry is missing	build.properties	/com.ibm.wala.ide	line 1	Plug-in Problem
'source.target/dependency/com.ibm.wala.core.jar' build entry is missing	build.properties	/com.ibm.wala.ide	line 1	Plug-in Problem
'source.target/dependency/com.ibm.wala.shrike.jar' build entry is missing	build.properties	/com.ibm.wala.ide	line 1	Plug-in Problem
'source.target/dependency/com.ibm.wala.util.jar' build entry is missing	build.properties	/com.ibm.wala.ide	line 1	Plug-in Problem

These errors are presumably associated with the mention of several target/dependency/com.ibm.wala.*.jar files in com.ibm.wala.ide/build.properties:

https://github.com/wala/WALA/blob/65e078d1cab52d6923bc7f982fca79185adce860/com.ibm.wala.ide/build.properties#L1-L7

Taking just one as an example, I cannot find any com.ibm.wala.cast.jar file anywhere in the imported WALA tree. If I run ./gradlew prepareMavenBuild publishToMavenLocal -x javadoc, I still find no com.ibm.wala.cast.jar files, though now there are is a com.ibm.wala.cast/build/libs/com.ibm.wala.cast-1.5.6-SNAPSHOT.jar file. And if I manually copy com.ibm.wala.cast/build/libs/com.ibm.wala.cast-1.5.6-SNAPSHOT.jar to com.ibm.wala.ide/target/dependency/com.ibm.wala.cast.jar, then the corresponding Plug-in problem goes away in Eclipse.

The Gradle changes in this pull request automate that process of copying jar files and stripping out the version information in their names. This is all triggered automatically whenever Eclipse Buildship synchronizes with Gradle, such as upon initial WALA import or whenever saving changes to any build.gradle file.

Is this a reasonable step toward fixing part of #742? It certainly doesn't fix all of it, but perhaps it is a step in the right direction?

Does automating this jar-copying task in Gradle make anything in com.ibm.wala.ide/pom.xml redundant? Or is that all unrelated?

[msridhar]

This is a good start! Couple things. This is the relevant pom.xml config for copying those jars:

https://github.com/wala/WALA/blob/a9157d8cce595798baffef04ae62e27a9e96c909/com.ibm.wala.ide/pom.xml#L24-L57

I think this could all be deleted if we get things working in the Gradle setup. Notice these two lines:

https://github.com/wala/WALA/blob/a9157d8cce595798baffef04ae62e27a9e96c909/com.ibm.wala.ide/pom.xml#L55-L56

The first sets the output directory target/dependency, and the second strips the version number, leading to jars like com.ibm.wala.cast.jar. I stripped versions just to make things less complicated in the Eclipse config when we are bumping versions.

The jars are referenced in two places in the Eclipse config. One is the MANIFEST.MF, e.g.:

https://github.com/wala/WALA/blob/a9157d8cce595798baffef04ae62e27a9e96c909/com.ibm.wala.ide/META-INF/MANIFEST.MF#L152-L156

The other is the build.properties, e.g.:

https://github.com/wala/WALA/blob/a9157d8cce595798baffef04ae62e27a9e96c909/com.ibm.wala.ide/build.properties#L4-L7

So hopefully that's helpful in understanding what is going on. We'd probably want to make a corresponding change for a couple of other ide projects that do this kind of jar copying. JSDT will be a bit more complex as it also copies 3rd-party dependencies:

https://github.com/wala/WALA/blob/169e5221df35c54571373ae5d1beb9c4689c6508/com.ibm.wala.ide.jsdt/pom.xml#L39-L52

Ideally, if/when we push through this change, maybe we should also stop using target/dependency as the destination folder; I just chose it based on copy-pasting. The target folder is where Maven puts its build artifacts, so running mvn clean will delete all these jars. mvn clean gets run often since AFAICT no Maven build ever succeeds without cleaning first 🙂 So if developers using these plug-ins are also running the Maven build, perhaps we should choose a different destination directory; something like lib should work fine. We just need to do a bunch of renames in the config files.

[liblit]

OK, it seems the entire configuration element from this subproject's pom.xml can be deleted. Would we also delete the preceding goals element?

https://github.com/wala/WALA/blob/a9157d8cce595798baffef04ae62e27a9e96c909/com.ibm.wala.ide/pom.xml#L21-L23

And if that goes away, does the entire build element go away too?

https://github.com/wala/WALA/blob/a9157d8cce595798baffef04ae62e27a9e96c909/com.ibm.wala.ide/pom.xml#L15-L62

We'd probably want to make a corresponding change for a couple of other ide projects that do this kind of jar copying. JSDT will be a bit more complex as it also copies 3rd-party dependencies

Yes, we can pull all of that logic into Gradle. Copying 3rd-party dependencies will work just fine: we already have an example in com.ibm.wala.dalvik/build.gradle of defining a custom sampleCup configuration

https://github.com/wala/WALA/blob/a9157d8cce595798baffef04ae62e27a9e96c909/com.ibm.wala.dalvik/build.gradle#L116-L118

adding one dependency to that configuration

https://github.com/wala/WALA/blob/a9157d8cce595798baffef04ae62e27a9e96c909/com.ibm.wala.dalvik/build.gradle#L128

then using the one jar file corresponding to that dependency as an input to some other task

https://github.com/wala/WALA/blob/a9157d8cce595798baffef04ae62e27a9e96c909/com.ibm.wala.dalvik/build.gradle#L97

A nice consequence of moving this into Gradle is that we will be closer to a "single source of truth" ideal in that Gradle, and only Gradle, will determine the versions of dependencies to use and how to resolve, download, and cache the corresponding jar files.

Ideally, if/when we push through this change, maybe we should also stop using target/dependency as the destination folder; I just chose it based on copy-pasting.

The Gradle-friendly place to put these would be somewhere under the corresponding subproject's build subdirectory. And that plants an interesting idea in my head. What if there were no "Maven build" at all, and instead Maven was just an external tool that Gradle runs as part of its build logic for certain specific tasks? After all, Gradle already runs many external tools for things that Gradle itself doesn't know how to do: gcc, shellcheck, etc. Gradle knows enough about dependencies to know which tools to run when, and perhaps it could be the same for Maven. There would be only one build system: Gradle. And it just so happens that some of Gradle's tasks are implemented in a way that they run Maven to get certain things done. But that's not something most WALA developers ever need to think about, beyond making sure that mvn is installed just like gcc or any other tool that Gradle will use to get its work done.

[msridhar]

And if that goes away, does the entire build element go away too?

Yes, I think we could delete the entire build element in this pom.xml.

We'd probably want to make a corresponding change for a couple of other ide projects that do this kind of jar copying. JSDT will be a bit more complex as it also copies 3rd-party dependencies

Yes, we can pull all of that logic into Gradle.

👍

The Gradle-friendly place to put these would be somewhere under the corresponding subproject's build subdirectory. And that plants an interesting idea in my head. What if there were no "Maven build" at all, and instead Maven was just an external tool that Gradle runs as part of its build logic for certain specific tasks? After all, Gradle already runs many external tools for things that Gradle itself doesn't know how to do: gcc, shellcheck, etc. Gradle knows enough about dependencies to know which tools to run when, and perhaps it could be the same for Maven. There would be only one build system: Gradle. And it just so happens that some of Gradle's tasks are implemented in a way that they run Maven to get certain things done. But that's not something most WALA developers ever need to think about, beyond making sure that mvn is installed just like gcc or any other tool that Gradle will use to get its work done.

This is a great idea! I love it. We would just need to call out to mvn as an executable to run the Eclipse Plug-In tests. The only thing is I don't know how well this fits in with Gradle sub-projects. It may be that this approach only works as a single task in the top-level build.gradle, rather than the preferable splitting of tasks among the sub-projects.

[liblit]

@msridhar, do you understand this compilation failure during the Maven build? Presumably the missing import should been provided by com.ibm.wala.core’s jar archive. And that should already have been installed under ~/.m2 by the Gradle publishToMavenLocal task. But what I’m not seeing is anything in com.ibm.wala.ide/pom.xml that tells Maven to look for a suitable com.ibm.wala.core jar archive in the first place. How would that dependency have been communicated to Maven before my changes? Is there something I should add to com.ibm.wala.ide/pom.xml now to tell Maven to look for a suitable com.ibm.wala.core jar archive in the local ~/.m2 repository that Gradle has already populated?

[msridhar]

Is the :com.ibm.wala.ide:eclipse task running as part of the Maven CI job? Right now this is what runs before the Maven build:

https://github.com/wala/WALA/blob/74bab0d97d36d9dc99a86c19659cf8ff0579fd8a/travis/before-install-maven#L3

Maybe we need to add the eclipse task there?

[liblit]

@msridhar wrote:

Is the :com.ibm.wala.ide:eclipse task running as part of the Maven CI job?

It is not, but I don't think it should need to be. At the point where that failure's happening, Maven isn't yet doing anything involving com.ibm.wala.ide/build.properties, so I wouldn't expect anything to be looking for the jar archives that :com.ibm.wala.ide:copyEclipsePluginDependencies installs under com.ibm.wala.ide/target/dependency. Maven is just trying to do basic Java compilation at this point, and everything it needs should be under ~/.m2. But I don't think we're telling Maven to look there. I suspect we should be listing some dependencies in com.ibm.wala.ide/pom.xml so that Maven will go find the required jar files under ~/.m2.

[msridhar]

Maven is doing compilation via Tycho:

https://travis-ci.org/github/liblit/WALA/jobs/679906421#L447

So it doesn't work like a standard Java compilation. It may well be using build.properties and MANIFEST.MF to find the dependencies, at least that's my guess.

[liblit]

OK, good point. I’ve added copying of those com.ibm.wala.ide/target/dependency files to prepareMavenBuild. Let’s see what happens! 🤞

[liblit]

Still fails. Different unresolved import, but still one that ought to have been satisfied by com.ibm.wala.core’s jar archive. I really do not know how dependency declaration and resolution happens in this Maven+Tycho environment.

[liblit]

Actually, the unresolved import we saw earlier still occurs now too. I think we’re just seeing the same errors in a different order. It’s not clear that copying jar archives into com.ibm.wala.ide/target/dependency before starting the Maven build had any effect.

[liblit]

I've now added the eclipse task to prepareMavenBuild too, both at the WALA root project as well as the com.ibm.wala.ide subproject. Local testing tells me that this will still fail in the same way.

I've been speculating that we needed to add dependency information to com.ibm.wala.ide/pom.xml to tell Maven to look for com.ibm.wala.core's jar archive, and presumably the same for cast, shrike, and util. However, the documentation for the eclipse-plugin packaging type being used here says:

Since Tycho uses OSGi bundle manifest to determine project dependencies, pom.xml file should NOT contain <dependency> section, and any dependencies inherited from parent project will be ignored by the build.

[msridhar]

Sigh. Nothing with Tycho is easy. I am busy but when I get time I can try to take a look. We can always try sticking in the WALA jars as regular Maven dependencies to see if that works, but it doesn't make sense.

[liblit]

Turn that frown upside-down, @msridhar! :upside_down_face: I figured out the missing dependency issue in 612aad2b4589211207bc367d016a18bf0d51f8dd. I didn't post an update here because I'd already been spamming you enough that evening as I tried various things. But the important part is that the tests are now passing, with multiple Maven-built projects having been converted to use Gradle to pre-position their dependencies.

[msridhar]

Turn that frown upside-down, @msridhar! 🙃 I figured out the missing dependency issue in 612aad2.

Woohoo! I guess it was a good idea to change the output directory 😄 Really excited to see this change working.

[msridhar]

This plugin to run Maven from Gradle might be useful:

https://github.com/dkorotych/gradle-maven-exec-plugin/blob/master/README.md

Haven’t tried it, just wanted to mention it

[msridhar]

@liblit I am wondering if we can close this PR? It's most likely quite out of date, and potentially not even needed anymore. No big deal if you want to keep it around though

[liblit]

Agreed. Perhaps obsolete, but certainly out-of-date.