KAFKA-14256: Upgrade from Scala 2.13.8 to 2.13.9

[mdedetrich]

This PR updates from Scala 2.13.8 to 2.13.9. Since its a binary compatible change there shouldn't be any problems.

Release notes can be found here https://github.com/scala/scala/releases/tag/v2.13.9

[mdedetrich]

So the update to Scala 2.13.9 seems to have found a regression in the scalac compiler, reported upstream https://github.com/scala/bug/issues/12648

[mdedetrich]

So I have updated the PR with the following changes

• Scala 2.13.10 just came out, so I bumped the Scala version (commit message has also been updated)
• I have added a workaround for the depreciation warning message in the commit Add workaround for change of behaviour in deprecation for 2.13.10.

The latest CI run has failed but it appears to be due to a possible unrelated problem, i.e. the zinc compiler has a timeout because kafka-streams-scala was compiling at the same time as core-scala

[2022-10-15T13:27:03.553Z] BUILD FAILED in 8m 24s

[2022-10-15T13:27:03.553Z] 237 actionable tasks: 192 executed, 45 up-to-date

[2022-10-15T13:27:03.553Z] 

[2022-10-15T13:27:03.553Z] See the profiling report at: file:///home/jenkins/workspace/Kafka_kafka-pr_PR-12675/build/reports/profile/profile-2022-10-15-13-18-44.html

[2022-10-15T13:27:03.553Z] A fine-grained performance profile is available: use the --scan option.

[2022-10-15T13:27:04.484Z] > Timeout waiting to lock zinc-1.6.1_2.13.10_11 compiler cache (/home/jenkins/.gradle/caches/7.5.1/zinc-1.6.1_2.13.10_11). It is currently in use by another Gradle instance.

[2022-10-15T13:27:04.484Z] Retrying due to zinc lock timeout

[2022-10-15T13:27:05.415Z] To honour the JVM settings for this build a single-use Daemon process will be forked. See https://docs.gradle.org/7.5.1/userguide/gradle_daemon.html#sec:disabling_the_daemon.

[2022-10-15T13:27:07.178Z] Daemon will be stopped at the end of the build 

@mimaison @cadonna @ijuma Would it be possible to retrigger the CI so I can see if this error is deterministic? If so I will have a look into the problem.

[mimaison]

I've kicked a new build: https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-12675/6/

[vjanelle]

Hmm, trying this locally with 2.8.2 (slightly forked), and the following test fails

> Task :core:integrationTest FAILED
kafka.admin.TopicCommandWithAdminClientTest.testDeleteTopicDoesNotRetryThrottlingQuotaExceededException() failed, log available in /Users/vjanelle/git/kafka/core/build/reports/testOutput/kafka.admin.TopicCommandWithAdminClientTest.testDeleteTopicDoesNotRetryThrottlingQuotaExceededException().test.stdout

TopicCommandWithAdminClientTest > testDeleteTopicDoesNotRetryThrottlingQuotaExceededException() FAILED
    Argument(s) are different! Wanted:
    admin.deleteTopics(
        [testDeleteTopicDoesNotRetryThrottlingQuotaExceededException-Qx8RDpByDZ],
        <custom argument matcher>
    );
    -> at kafka.admin.TopicCommandWithAdminClientTest.testDeleteTopicDoesNotRetryThrottlingQuotaExceededException(TopicCommandWithAdminClientTest.scala:779)
    Actual invocations have different arguments:
    admin.listTopics(
        org.apache.kafka.clients.admin.ListTopicsOptions@669fa03e
    );
    -> at kafka.admin.TopicCommand$AdminClientTopicService.getTopics(TopicCommand.scala:368)
    admin.deleteTopics(
        [testDeleteTopicDoesNotRetryThrottlingQuotaExceededException-Qx8RDpByDZ],
        org.apache.kafka.clients.admin.DeleteTopicsOptions@4a7bb32e
    );
    -> at kafka.admin.TopicCommand$AdminClientTopicService.deleteTopic(TopicCommand.scala:360)
        at kafka.admin.TopicCommandWithAdminClientTest.testDeleteTopicDoesNotRetryThrottlingQuotaExceededException(TopicCommandWithAdminClientTest.scala:779)

I haven't confirmed with 3.x yet though.

[mdedetrich]

So I managed to solve one problem but created another. So I have solved the issues with the zinc incremental compiler (it appears that the version of zinc was too old for the new Scala compiler being used) however now there is a problem of the where the rat task is complaining that an incompatible license was brought in ????

I cannot produce the problem locally at all either, does anyone know if its possible to grab the report file directly from the CI so I can see what it's complaining about?

Hmm, trying this locally with 2.8.2 (slightly forked), and the following test fails

@vjanelle Can you setup a branch setup with your exact setup, I can then have a look into it.

[vjanelle]

@mdedetrich https://github.com/vjanelle/kafka/tree/2.8.2-scalaupdate - thank you 🙇

[omkreddy]

@mdedetrich Thanks for the PR. I have run this PR with rat verbose output bug fix

Results are (https://github.com/apache/kafka/pull/12785, https://github.com/apache/kafka/runs/9090714417): I can see be below rat task error.

Files with unapproved licenses:
 - /home/jenkins/workspace/Kafka_kafka-pr_PR-12785/buildoutput.log

compilation errors with JDK 11 and Scala 2.13

Argument(s) are different! Wanted:
admin.deleteTopics(
    [topicName],
    <custom argument matcher>
);
-> at kafka.admin.TopicCommandTest.testDeleteTopicDoesNotRetryThrottlingQuotaExceededException(TopicCommandTest.scala:207)
Actual invocations have different arguments:
admin.listTopics(
    ListTopicsOptions(listInternal=true)
);
-> at kafka.admin.TopicCommand$TopicService.getTopics(TopicCommand.scala:370)
admin.deleteTopics(
    [topicName],
    org.apache.kafka.clients.admin.DeleteTopicsOptions@bd09a26
);
-> at kafka.admin.TopicCommand$TopicService.deleteTopic(TopicCommand.scala:362)

[omkreddy]

@mdedetrich any update here? let us know if you need any help.

[mdedetrich]

I was busy last week so I didn't have time for it, will try to look later this week

[egyedt]

The following dependency should also be checked: scoverage: "1.4.11" The 2.0.7 version of it, is the first one where Scala 2.13.10 is supported. https://mvnrepository.com/artifact/org.scoverage/scalac-scoverage-plugin_2.13.10/2.0.7 https://github.com/scoverage/scalac-scoverage-plugin/releases

[vjanelle]

@mdedetrich looks like confluent has something in their branch linked to here that would work for the arg matching issue.

[ijuma]

I rebased the branch and fixed the failing test.

[ijuma]

Java 8 and 11 passed. Java 17 had a few flaky unrelated failures:

Build / JDK 17 and Scala 2.13 / org.apache.kafka.common.network.SslVersionsTransportLayerTest.tlsServerProtocol = [TLSv1.3, TLSv1.2], tlsClientProtocol = [TLSv1.3] Build / JDK 17 and Scala 2.13 / org.apache.kafka.common.record.MemoryRecordsBuilderTest.[6] magic=2, bufferOffset=0, compressionType=gzip Build / JDK 17 and Scala 2.13 / org.apache.kafka.common.record.MemoryRecordsBuilderTest.[18] magic=1, bufferOffset=15, compressionType=gzip