Add Gradle generator

[hadess]

I don't know what I'm doing, but I can parse the logs of a Gradle build and generate a json file to build Ghidra.

[mpi3d]

The script doesn't work out of the box. I had to make some changes: flatpak-gradle-generator.zip Also it supports .exe deps

[hadess]

The script doesn't work out of the box. I had to make some changes:

Could you please post a new version of this without changing all the quotes by double-quotes? It adds a lot of changes that aren't necessary to implement what you need.

I also don't understand why you removed the regexp. Maybe the easiest would be if you attached the full gradle log for us to test, then we can try it out.

[mpi3d]

Here it is: files.zip

[hadess]

I kept the code to create arch-specific urls, but I also:

• changed the arguments parsing to match the help (it wasn't possible to request multiple arches without changing the order or arguments in a way that didn't match the help output)
• used flatpak arches instead of the "gradle" ones in the arguments
• modified regex to not catch local filenames (instead of expecting "Downloading" etc.)
• add only-arches info to urls that need it

I'm pretty happy with this now, so if anyone else wants to experiment.

[mpi3d]

Good. Any idea when this will be merged?

[hadess]

Good. Any idea when this will be merged?

No, but porting the jadx or Bisq Flatpaks to use it would certainly show whether it's fit for purpose. In the meanwhile, it's not disappearing, and we can use it for Ghidra.

[mpi3d]

Ok

[Mailaender]

I couldn't get the script to work, but I don't understand the approach. For @terasology I can ./gradlew dependencies which gives me:

Root project 'Terasology'
------------------------------------------------------------

api (n)
No dependencies

apiElements (n)
No dependencies

archives - Configuration for archive artifacts. (n)
No dependencies

classpath
No dependencies

codeMetrics
\--- org.terasology.config:codemetrics:1.6.3

default - Configuration for default artifacts. (n)
No dependencies

enforcedApiElements (n)
No dependencies

enforcedRuntimeElements (n)
No dependencies

natives
+--- org.lwjgl:lwjgl-bom:3.2.3
|    +--- org.lwjgl:lwjgl:3.2.3 (c)
|    +--- org.lwjgl:lwjgl-assimp:3.2.3 (c)
|    +--- org.lwjgl:lwjgl-glfw:3.2.3 (c)
|    +--- org.lwjgl:lwjgl-openal:3.2.3 (c)
|    +--- org.lwjgl:lwjgl-opengl:3.2.3 (c)
|    \--- org.lwjgl:lwjgl-stb:3.2.3 (c)
+--- org.lwjgl:lwjgl -> 3.2.3
+--- org.lwjgl:lwjgl-assimp -> 3.2.3
|    \--- org.lwjgl:lwjgl:3.2.3
+--- org.lwjgl:lwjgl-glfw -> 3.2.3
|    \--- org.lwjgl:lwjgl:3.2.3
+--- org.lwjgl:lwjgl-openal -> 3.2.3
|    \--- org.lwjgl:lwjgl:3.2.3
+--- org.lwjgl:lwjgl-opengl -> 3.2.3
|    \--- org.lwjgl:lwjgl:3.2.3
+--- org.lwjgl:lwjgl-stb -> 3.2.3
|    \--- org.lwjgl:lwjgl:3.2.3
+--- org.terasology.jnlua:jnlua_natives:0.1.0-SNAPSHOT
\--- org.terasology.jnbullet:JNBullet:1.0.2

Can this be parsed instead?

[mpi3d]

I tried on Ghidra. But it doesn't list all the dependencies.

[hadess]

I couldn't get the script to work, but I don't understand the approach.

What failed exactly? Can you provide the gradle log?

[Mailaender]

I am missing Python dependencies on @openSUSE.

[mpi3d]

Which ones? aiohttp? Just do a pip install aiohttp

[Mailaender]

Seems like updating pip with pip itself solved the installation problems. Yet the script does not run:

Traceback (most recent call last):
  File "./flatpak-gradle-generator.py", line 109, in <module>
    main()
  File "./flatpak-gradle-generator.py", line 101, in main
    sources = asyncio.run(parse_urls(urls, urls_arch, args.destdir))
AttributeError: module 'asyncio' has no attribute 'run'

[gasinvein]

Seems like updating pip with pip itself solved the installation problems. Yet the script does not run:

Traceback (most recent call last):
  File "./flatpak-gradle-generator.py", line 109, in <module>
    main()
  File "./flatpak-gradle-generator.py", line 101, in main
    sources = asyncio.run(parse_urls(urls, urls_arch, args.destdir))
AttributeError: module 'asyncio' has no attribute 'run'

This means your Python version is way too old. IIRC asyncio.run was added in 3.7.

[hadess]

AttributeError: module 'asyncio' has no attribute 'run'

From https://docs.python.org/3/library/asyncio-task.html#asyncio.run

asyncio.run(coro, *, debug=False) New in version 3.7.

Python 3.7 was released in 2018, close to 4 years ago. Time to update.

[TobTobXX]

Just for some external input: I've also worked on this and have a semi-working script here: https://github.com/TobTobXX/flatpak-builder-tools/blob/master/gradle/flatpak-gradle-generator.sh.

Parsing build logs or ./gradlew dependencies both have the issue that they don't include dependencies that are pure POMs (ie. parent POMs that are used for inheritance and referenced external POMs).

Instead, my script runs a ./gradle build --dry-run, but sets a custom cache directory. This ensures all assets that are needed for an offline build are downloaded. (One issue my script has to deal with now is that some POMs differ depending on the mirror you use.)

I've not submitted a PR yet, since I'm still testing my script on various projects, but if you want to use it, go ahead.

[hadess]

Parsing build logs or ./gradlew dependencies both have the issue that they don't include dependencies that are pure POMs (ie. parent POMs that are used for inheritance and referenced external POMs).

I think this might be a problem with using gradlew instead of gradle, or a problem with the build scripts because the build tools should already log all the downloaded assets.

But not knowing what software you're trying to package, or the exact steps, it's difficult to say. Note that the build script we use for Ghidra uses a sandboxed dependency download section, to avoid any sort of cache from being used inadvertently.

I've not submitted a PR yet, since I'm still testing my script on various projects, but if you want to use it, go ahead.

I doubt that a bash script would be accepted. In any case, that would be something to discuss on your own PR, not this one.

[Mailaender]

Can you maybe post an example usage? I managed to download all dependencies for JSkat, but it seems they are not discovered by Gradle:

Plugin [id: 'org.openjfx.javafxplugin', version: '0.0.13'] was not found in any of the following sources:

- Gradle Core Plugins (plugin is not in 'org.gradle' namespace)
- Plugin Repositories (could not resolve plugin artifact 'org.openjfx.javafxplugin:org.openjfx.javafxplugin.gradle.plugin:0.0.13')
  Searched in the following repositories:
    Gradle Central Plugin Repository

[hadess]

There's already an explanation in the README added in the patch. But I tested the generator with https://github.com/b0n541/jskat-multimodule (next time, please make sure to attach the full output of what you did, here I had to even search what jskat was).

$ flatpak install -y --user org.freedesktop.Sdk.Extension.openjdk17//21.08
[...]
$ flatpak run --command=bash --share=network --filesystem=`pwd` -d org.freedesktop.Sdk//21.08
[📦 org.freedesktop.Sdk jskat-multimodule]$ source /usr/lib/sdk/openjdk17/enable.sh
[📦 org.freedesktop.Sdk jskat-multimodule]$ ./gradlew -g gradle-cache/ --info --console plain clean installDist > gradle-log.txt

Running the script on the log threw an error, but I edited the script to remove the URL that caused the error, and re-ran it, and it generated a json file. It didn't seem like the file was needed, as it built without it.

[Mailaender]

Running the generator is not a problem, but I don't know how to apply this in the Flatpak manifest.

[hadess]

Running the generator is not a problem, but I don't know how to apply this in the Flatpak manifest.

It's a json file, you include it. See the Ghidra manifest for example: https://github.com/flathub/org.ghidra_sre.Ghidra/blob/master/org.ghidra_sre.Ghidra.json#L142

[Mailaender]

The shell script moves files by default into maven-local where Gradle can't find it.

[MessiasLima]

Does it need to be merged in order to use it?

[hadess]

The shell script moves files by default into maven-local where Gradle can't find it.

I'm not sure where you get that, there's no mentions of maven-local in the script.

Does it need to be merged in order to use it?

Absolutely not, a second user might make it easier to get this patch merged, hopefully.

[MessiasLima]

I tried to run in my application and I'm using detekt for static code analysis. However, when I run it cannot get the .jar file from detekt because there is no .jar file in the repo. As you can see here.

The script fails because of that but detekt is not even used in production code.

Do you have any hint about what to do in this case?

[mpi3d]

You might implement an option in the script to use a custom url for specific packages. Or even ignore the url.

Or you can currently remove the url manually.

[MessiasLima]

You might implement an option in the script to use a custom url for specific packages. Or even ignore the url.

Or you can currently remove the url manually.

I ignored the Url and it worked, thank you!

[hfiguiere]

I'd be fine merging this. It works for you then it can be fixed later for other use cases that fail. Otherwise it never will.

[TingPing]

Please have somebody in CODEOWNERS, otherwise :+1:

[hadess]

Please have somebody in CODEOWNERS, otherwise 👍

Done.