Improve compatibility with the Gradle API, and follow best practices

[aSemy]

trafficstars

Is your feature request related to a problem? Please describe

Hi, this is related to the Dokka Gradle plugin, which I would like to help improve.

Improving usage of Gradle would help with the following issues:

• https://github.com/Kotlin/dokka/issues/2231
• https://github.com/Kotlin/dokka/issues/2515
• https://github.com/Kotlin/dokka/issues/1833
• https://github.com/Kotlin/dokka/issues/1864
• https://github.com/Kotlin/dokka/issues/1752
• https://github.com/Kotlin/dokka/issues/2612
• https://github.com/Kotlin/dokka/issues/1217
• https://github.com/Kotlin/dokka/issues/1382
• https://github.com/Kotlin/dokka/issues/2286
• https://github.com/Kotlin/dokka/pull/830
• https://github.com/Kotlin/dokka/issues/2822
• https://github.com/Kotlin/dokka/pull/2756
• https://github.com/Kotlin/dokka/issues/2903

These PRs are related:

• https://github.com/Kotlin/dokka/pull/2499
• https://github.com/Kotlin/dokka/pull/2799
• https://github.com/Kotlin/dokka/pull/2807
• https://github.com/Kotlin/dokka/pull/2816

Describe the solution you'd like

• use Kotlin DSL helpers to make the code more Kotlin-esque, for example https://github.com/Kotlin/dokka/blob/83174361becb2af227159834cdf6e14db9300c53/runners/gradle-plugin/src/main/kotlin/org/jetbrains/dokka/gradle/gradleConfigurations.kt#L10
• Tasks that require downloading files should cache and re-use files https://github.com/Kotlin/dokka/issues/2695
• Use configurations for sharing outputs between projects https://docs.gradle.org/7.3/samples/sample_jvm_multi_project_with_code_coverage.html
• Remove non-idiomatic helper functions https://github.com/Kotlin/dokka/blob/master/runners/gradle-plugin/src/main/kotlin/org/jetbrains/dokka/gradle/DokkaProperty.kt
• Rename files to match the contents, e.g. https://github.com/Kotlin/dokka/blob/master/runners/gradle-plugin/src/main/kotlin/org/jetbrains/dokka/gradle/main.kt
• Remove usages of internal Gradle API classes https://github.com/Kotlin/dokka/blob/83174361becb2af227159834cdf6e14db9300c53/runners/gradle-plugin/src/main/kotlin/org/jetbrains/dokka/gradle/AbstractDokkaLeafTask.kt#L16
• avoid using allprojects {} or subprojects {} https://github.com/Kotlin/dokka/blob/83174361becb2af227159834cdf6e14db9300c53/runners/gradle-plugin/src/test/kotlin/org/jetbrains/dokka/gradle/AbstractDokkaParentTaskTest.kt#L21-L23 https://github.com/Kotlin/dokka/blob/83174361becb2af227159834cdf6e14db9300c53/runners/gradle-plugin/src/main/kotlin/org/jetbrains/dokka/gradle/AbstractDokkaParentTask.kt#L54-L60

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered

Additional context

Perhaps the dokka team could create a 'gradle-plugin' tag for all relevant GitHub issues? https://github.com/Kotlin/dokka/issues?q=is%3Aissue+is%3Aopen++gradle

Are you willing to provide a PR?

Yes. I will try and split this up into smaller PRs.

[IgnatBeresnev]

Wow... thanks! Our Gradle plugin and everything around it definitely needs work. I'll take us some time to go through PRs, so please bear with us :sweat_smile:

[aSemy]

So I've submitted a few PRs and they make some progress towards improving the plugin, but actually started re-implementing the plugin mostly to get my head around it, but also I think it would be nice to have a fresh start. I see there's code in the plugin from 2015, and Gradle has changed a lot since then.

Here's what I've got so far:

• Use the dokka-cli project to execute Dokka
• use javaexec {} to run the cli, not reflection magic.
• (Aside: because of this the Gradle plugin could be made completely independent from the Dokka code base, with the dokka-cli version being customisable, perhaps with a runtime check for the latest version. This decoupling is used in other plugins, like JaCoCo, gradle-intellij-plugin.)
• Simplify the Dokka tasks to three: 2 for configuration, 1 for executing. These use file-based outputs to share configuration between projects, ensuring tasks can be cached properly:
  1. DokkaConfigurationTask - takes user/project input and creates a Dokka JSON config file, to pass to dokka-cli. It also consumes DokkaModuleDescriptions from subprojects (using Gradle configurations)
  2. DokkaModuleConfigurationTask - creates a DokkaModuleDescription subprojects, and encodes it as a JSON file, which can be shared properly with Gradle configurations.
  3. DokkaExecutorTask - consumes the configuration file produced by DokkaConfigurationTask, uses the file to correctly mark task inputs/outputs, and runs Dokka using dokka-cli + javaexec{}
• Create Dokka specific Gradle Configurations to share the JSON config files between subprojects (based on this Gradle example: https://docs.gradle.org/7.3/samples/sample_jvm_multi_project_with_code_coverage.html)

I'm not sure how much impact this will have. Perhaps it's possible to 'backport' it to the current plugin, or maybe it needs a clean break. It's still a work-in-progress. I can make a PR if you want to have a very early look?

[Goooler]

#2652 would be a part of this?

[Goooler]

And CC is WIP in #2499.

[aSemy]

#2652 would be a part of this?

#2652 is about improving the Gradle build scripts for the Dokka project, not the Dokka Gradle plugin. They both use Gradle, but they are independent from one another.

And CC is WIP in #2499.

Oh good catch, I missed that PR. That PR is indeed related, and should help.

[IgnatBeresnev]

Hi! Sorry it's taking so long, I've only now finally got time after re-writing documentation for stable release.

I'll go over all Gradle PRs in the upcoming weeks :+1:

[IgnatBeresnev]

If you don't mind, I'll use this issue as an umbrella issue to cover all Gradle-related PRs from you and @Goooler. I'll include "infrastructure" tagged issues in my comment as well as they're about following Gradle's best practices and there aren't that many of them.

Below I'll write down the order in which the PRs should be reviewed and merged, feel free to correct me.

First wave

Can be reviewed now, but merged after we branch off for 1.8.10: :heavy_check_mark: ~* #2833~ :heavy_check_mark: ~* #2707~ :heavy_check_mark: ~* #2705~ :heavy_check_mark: ~* (infra) #2704~

:heavy_check_mark: ~Blocked by #2841 (ETA: right after 1.8.10):~ :heavy_check_mark: ~* (approved) #2702~

:heavy_check_mark: ~Blocked by #2704 (ETA: right after 1.8.10)~ :heavy_check_mark: ~* (infra) #2912~

:heavy_check_mark: ~Blocked by #2912~ :heavy_check_mark: ~* (infra) #2884~

Second wave

• (needs a solution) #2499
• (we have some concerns to discuss) #2740

Third wave:

• #2839

To be discussed further / postponed:

• (might not be needed) #2759

[aajtodd]

What's the status of this? It looks like a lot of great improvements that we would like to see/use (gradle worker API, better cache support, fix inter project task dependency relationships, etc).

The AWS SDK for Kotlin is an abnormally large project (300+ modules). It takes several hours to build everything and we are weighing our options on how to get this time down (including possibly investigating moving to https://github.com/adamko-dev/dokkatoo).

There are also some issues with upgrading to Gradle 8.2+ w/Dokka 1.8+ that we are wondering might be fixed by some of these changes.

e.g.

    Reason: Task ':services:dokkaHtmlMultiModule' uses this output of task ':dokkaHtmlMultiModule' without declaring an explicit or implicit dependency. This can lead
 to incorrect results being produced, depending on what order the tasks are executed.  

[IgnatBeresnev]

@aajtodd I definitely agree, and we honestly can't wait ourselves to address Gradle problems, but our resources have been spread thin, and there have been multiple things falling apart at the same time (K2/analysis-related), so we couldn't focus on Gradle much.

Rest assured, we understand that our Gradle plugin is a severe problem for Dokka's users, and it is very high on our list. We should hopefully get to it quite soon - likely within the next month.

In the meantime, if migrating to github.com/adamko-dev/dokkatoo resolves your problems - please, do use it :) Adam did an amazing job, and we plan to merge dokkatoo into Dokka, making it Dokka's new Gradle plugin - albeit, with some configuration DSL changes, so expect some migration work up ahead. It would also help us with finding bugs early, so you'd be helping us with testing.

P.S., Dokka 1.9.0 should work with aws-sdk-kotlin, we've somewhat "fixed" the single-target multiplatform support (with your help in debugging), and I'm still trying to get to providing more details on this in #2679 - will try hard to do it this friday or next week.