kafka KAFKA-15062: Adding ppc64le build stage

More detailed description of your change, if necessary. The PR title and PR message become the squashed commit message, so use a separate comment to ping reviewers.

Summary of testing strategy (including rationale) for the feature or bug fix. Unit and/or integration tests are expected for any behaviour change and system tests should be considered for larger changes.

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Jun 06 '23 10:06 Vaibhav-Nazare

@dajac Can you check the please check the PR.

Jun 07 '23 05:06 Vaibhav-Nazare

Hi @Vaibhav-Nazare It was intentionally disabled in the past since the infrastructure was flaky https://github.com/apache/kafka/pull/12380 If we want to officially test and support this platform, I would suggest that you start a KIP similar to https://cwiki.apache.org/confluence/display/KAFKA/KIP-826%3A+Define+platforms+supported and initiate a discussion in the community about it.

Jun 07 '23 10:06 divijvaidya

Hi @divijvaidya Thanks for the information, will start a KIP.

Also FYI new power VMs are created/managed by ASF infra and much more stable https://issues.apache.org/jira/browse/INFRA-24663

Jun 07 '23 10:06 Vaibhav-Nazare

Hi @divijvaidya I have created the KIP https://cwiki.apache.org/confluence/display/KAFKA/KIP-942%3A+Add+Power%28ppc64le%29+support Please let us know the further path forward.

Jun 19 '23 06:06 Vaibhav-Nazare

Hey @Vaibhav-Nazare The next step us to start a [DISCUSS] thread in the mailing list where the community can have a discussion on the KIP. The process of starting a KIP is mentioned here: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-Process

Jun 19 '23 09:06 divijvaidya

HI @divijvaidya Thanks for the information , I have created a discussion thread for the KIP as per given process https://lists.apache.org/thread/y62m18k5bgsbcmkw2f80vwnj2vjgy7nz

Jun 19 '23 10:06 Vaibhav-Nazare

Hi @divijvaidya any further updates for us?

Jun 23 '23 05:06 Vaibhav-Nazare

Hey @Vaibhav-Nazare A KIP needs at least 3 committer votes and I believe we haven't heard from other folks in the community on the KIP. I am waiting for others to chime in. Otherwise, we can start a vote in 2 weeks (and hope that the KIP gets at least 3 votes).

Jun 27 '23 19:06 divijvaidya

Hi @divijvaidya Any further updates on nightly job enablement?

Jul 16 '23 15:07 Vaibhav-Nazare

Hi @cmccabe @mimaison @divijvaidya Any further updates on nightly job enablement?

Jul 20 '23 07:07 Vaibhav-Nazare

@Vaibhav-Nazare The KIP needs to be voted. So far you've only started a discussion. I'd recommend replying to the discussion thread to ask for any more feedback, otherwise you can start a vote in the next few days. See https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals for the process

Jul 20 '23 07:07 mimaison

Hi @mimaison @cmccabe If you agree can we have your votes too on the KIP please, @divijvaidya has already voted.

Aug 21 '23 03:08 Vaibhav-Nazare

Hi @divijvaidya @mimaison @cmccabe can we look into this PR as the KIP-942 has been adopted

Sep 21 '23 12:09 Vaibhav-Nazare

Hi @divijvaidya @mimaison @cmccabe any updates on how we want to proceed?

Sep 29 '23 04:09 Vaibhav-Nazare

Thanks @Vaibhav-Nazare for the KP and PR.

I've not looked at Jenkinsfiles in a while, can you clarify why you used nested stage fields unlike the other existing platforms?

Oct 09 '23 10:10 mimaison

Hi @mimaison, thanks for the reply. We were referring to older Jenkins file before the support was disabled on Power, No specific reason to use nested stage fields.

If any modification is required, would do it according to accepted and correct ways, please do let me know.

Oct 10 '23 13:10 Vaibhav-Nazare

Then I'd probably make the new ppc job copy the existing jobs and use a single stage.

Another question: What will be the process of updating the Java version? Can this be part of the config or will we also need to contact the Apache Infra team?

Oct 11 '23 09:10 mimaison

@mimaison As my point of view for Java version , may be we can have this part of config itself so we don't have to raise tickets frequently to ASF team, wdyt?

Also are you going to setup a seprate ppc job at https://ci-builds.apache.org/job/Kafka/ ?

Oct 11 '23 11:10 Vaibhav-Nazare

Yes it would be better if the Java version can be specified as a config. It's something you need to figure out.

The job will be automatically created when we commit the updated Jenkinsfile.

Oct 11 '23 14:10 mimaison

Hi @mimaison I have updated the stage for PowerPC as of now, for the Java Version upgrade I need to figure out how to do it, may be we can have it in another PR?

Oct 12 '23 07:10 Vaibhav-Nazare

Thanks for the updates! Can we switch to jdk_21_latest? as it's the newest java version we use in the other stages

Nov 08 '23 16:11 mimaison

So obvious reasons, changes in Jenkinsfile are only tested if the PR is from a committer. So I opened https://github.com/apache/kafka/pull/14717 to run your changes.

You can see the Power build failed immediately with:

ERROR: JAVA_HOME is set to an invalid directory: /home/jenkins/tools/java/latest21
Please set the JAVA_HOME variable in your environment to match the
location of your Java installation.

Nov 08 '23 16:11 mimaison

Raised a JIRA ticket to have jdk_21_latest installed on shared power nodes https://issues.apache.org/jira/browse/INFRA-25149

Nov 08 '23 17:11 Vaibhav-Nazare

Hi @mimaison Can we rerun the job now, jdk_21_latest has been installed on the shared power nodes by ASF infra.

Nov 13 '23 05:11 Vaibhav-Nazare

Rerunning the build: https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14717/2/pipeline/14/

Nov 13 '23 09:11 mimaison

Hi @mimaison I ran the kafka.zk.ZkMigrationIntegrationTest locally and all the related tests are passing.

./gradlew -PscalaVersion=2.13.12 core:test --tests kafka.zk.ZkMigrationIntegrationTest --profile --continue -PkeepAliveMode="session"
-PtestLoggingEvents=started,passed,skipped,failed -PignoreFailures=true -PmaxParallelForks=2 -PmaxTestRetries=1 -PmaxTestRetryFailures=10
Starting a Gradle Daemon, 1 busy Daemon could not be reused, use --status for details

> Configure project :
Starting build with version 3.7.0-SNAPSHOT (commit id e7f4f5df) using Gradle 8.3, Java 21 and Scala 2.13.12
Build properties: maxParallelForks=2, maxScalacThreads=8, maxTestRetries=1

> Task :storage:storage-api:processResources NO-SOURCE
> Task :server-common:processResources NO-SOURCE
> Task :storage:storage-api:processTestResources NO-SOURCE
> Task :group-coordinator:processResources UP-TO-DATE
> Task :raft:processResources UP-TO-DATE
> Task :metadata:processResources UP-TO-DATE
> Task :core:processResources UP-TO-DATE
> Task :server-common:processTestResources UP-TO-DATE
> Task :server-common:createVersionFile UP-TO-DATE
> Task :group-coordinator:processTestResources UP-TO-DATE
> Task :metadata:processTestResources UP-TO-DATE
> Task :core:processTestResources UP-TO-DATE
> Task :tools:tools-api:processResources NO-SOURCE
> Task :storage:storage-api:createVersionFile UP-TO-DATE
> Task :storage:processResources UP-TO-DATE
> Task :raft:processTestResources UP-TO-DATE
> Task :tools:tools-api:createVersionFile UP-TO-DATE
> Task :storage:createVersionFile UP-TO-DATE
> Task :raft:createVersionFile UP-TO-DATE
> Task :clients:processResources UP-TO-DATE
> Task :clients:createVersionFile UP-TO-DATE
> Task :generator:compileJava UP-TO-DATE
> Task :generator:processResources NO-SOURCE
> Task :generator:classes UP-TO-DATE
> Task :clients:processTestResources UP-TO-DATE
> Task :generator:jar UP-TO-DATE
> Task :core:processMessages UP-TO-DATE
> Task :group-coordinator:processMessages UP-TO-DATE
> Task :metadata:processMessages UP-TO-DATE
> Task :raft:processMessages UP-TO-DATE
> Task :storage:processMessages UP-TO-DATE
> Task :clients:processMessages UP-TO-DATE
> Task :clients:compileJava UP-TO-DATE
> Task :clients:classes UP-TO-DATE
> Task :clients:processTestMessages UP-TO-DATE
> Task :tools:tools-api:compileJava UP-TO-DATE
> Task :tools:tools-api:classes UP-TO-DATE
> Task :tools:tools-api:jar UP-TO-DATE
> Task :server-common:compileJava UP-TO-DATE
> Task :server-common:classes UP-TO-DATE
> Task :server-common:jar UP-TO-DATE
> Task :storage:storage-api:compileJava UP-TO-DATE
> Task :storage:storage-api:classes UP-TO-DATE
> Task :raft:compileJava UP-TO-DATE
> Task :storage:storage-api:jar UP-TO-DATE
> Task :raft:classes UP-TO-DATE
> Task :raft:jar UP-TO-DATE
> Task :storage:compileJava UP-TO-DATE
> Task :storage:classes UP-TO-DATE
> Task :storage:jar UP-TO-DATE
> Task :metadata:compileJava UP-TO-DATE
> Task :metadata:classes UP-TO-DATE
> Task :metadata:jar UP-TO-DATE
> Task :clients:compileTestJava UP-TO-DATE
> Task :group-coordinator:compileJava UP-TO-DATE
> Task :group-coordinator:classes UP-TO-DATE
> Task :core:compileJava NO-SOURCE
> Task :group-coordinator:jar UP-TO-DATE
> Task :clients:shadowJar UP-TO-DATE
> Task :clients:jar SKIPPED
> Task :clients:testClasses UP-TO-DATE
> Task :storage:storage-api:compileTestJava UP-TO-DATE
> Task :storage:storage-api:testClasses UP-TO-DATE
> Task :server-common:compileTestJava UP-TO-DATE
> Task :server-common:testClasses UP-TO-DATE
> Task :raft:compileTestJava UP-TO-DATE
> Task :raft:testClasses UP-TO-DATE
> Task :group-coordinator:compileTestJava UP-TO-DATE
> Task :group-coordinator:testClasses UP-TO-DATE
> Task :metadata:compileTestJava UP-TO-DATE
> Task :metadata:testClasses UP-TO-DATE
> Task :core:compileScala UP-TO-DATE
> Task :core:classes UP-TO-DATE
> Task :core:checkstyleMain UP-TO-DATE
> Task :core:compileTestJava NO-SOURCE
> Task :core:compileTestScala UP-TO-DATE
> Task :core:testClasses UP-TO-DATE
> Task :core:checkstyleTest UP-TO-DATE
> Task :core:spotbugsMain UP-TO-DATE

> Task :core:test

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > [1] Type=ZK, MetadataVersion=3.4-IV0, Security=PLAINTEXT STARTED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > [1] Type=ZK, MetadataVersion=3.4-IV0, Security=PLAINTEXT PASSED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > [2] Type=ZK, MetadataVersion=3.5-IV2, Security=PLAINTEXT STARTED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > [2] Type=ZK, MetadataVersion=3.5-IV2, Security=PLAINTEXT PASSED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > [3] Type=ZK, MetadataVersion=3.6-IV2, Security=PLAINTEXT STARTED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > [3] Type=ZK, MetadataVersion=3.6-IV2, Security=PLAINTEXT PASSED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > [4] Type=ZK, MetadataVersion=3.7-IV0, Security=PLAINTEXT STARTED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > [4] Type=ZK, MetadataVersion=3.7-IV0, Security=PLAINTEXT SKIPPED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > [5] Type=ZK, MetadataVersion=3.7-IV1, Security=PLAINTEXT STARTED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > [5] Type=ZK, MetadataVersion=3.7-IV1, Security=PLAINTEXT PASSED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testDualWriteScram(ClusterInstance) > [1] Type=ZK, Name=testDualWriteScram, MetadataVersion=3.5-IV2, Security=PLAINTEXT STARTED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testDualWriteScram(ClusterInstance) > [1] Type=ZK, Name=testDualWriteScram, MetadataVersion=3.5-IV2, Security=PLAINTEXT PASSED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testNewAndChangedTopicsInDualWrite(ClusterInstance) > [1] Type=ZK, Name=testNewAndChangedTopicsInDualWrite, MetadataVersion=3.4-IV0, Security=PLAINTEXT STARTED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testNewAndChangedTopicsInDualWrite(ClusterInstance) > [1] Type=ZK, Name=testNewAndChangedTopicsInDualWrite, MetadataVersion=3.4-IV0, Security=PLAINTEXT PASSED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testDualWriteQuotaAndScram(ClusterInstance) > [1] Type=ZK, Name=testDualWriteQuotaAndScram, MetadataVersion=3.5-IV2, Security=PLAINTEXT STARTED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testDualWriteQuotaAndScram(ClusterInstance) > [1] Type=ZK, Name=testDualWriteQuotaAndScram, MetadataVersion=3.5-IV2, Security=PLAINTEXT PASSED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testMigrate(ClusterInstance) > [1] Type=ZK, Name=testMigrate, MetadataVersion=3.4-IV0, Security=PLAINTEXT STARTED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testMigrate(ClusterInstance) > [1] Type=ZK, Name=testMigrate, MetadataVersion=3.4-IV0, Security=PLAINTEXT PASSED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testMigrateAcls(ClusterInstance) > [1] Type=ZK, Name=testMigrateAcls, MetadataVersion=3.4-IV0, Security=PLAINTEXT STARTED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testMigrateAcls(ClusterInstance) > [1] Type=ZK, Name=testMigrateAcls, MetadataVersion=3.4-IV0, Security=PLAINTEXT PASSED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testStartZkBrokerWithAuthorizer(ClusterInstance) > [1] Type=ZK, Name=testStartZkBrokerWithAuthorizer, MetadataVersion=3.4-IV0, Security=PLAINTEXT STARTED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testStartZkBrokerWithAuthorizer(ClusterInstance) > [1] Type=ZK, Name=testStartZkBrokerWithAuthorizer, MetadataVersion=3.4-IV0, Security=PLAINTEXT PASSED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testDualWrite(ClusterInstance) > [1] Type=ZK, MetadataVersion=3.4-IV0, Security=PLAINTEXT STARTED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testDualWrite(ClusterInstance) > [1] Type=ZK, MetadataVersion=3.4-IV0, Security=PLAINTEXT PASSED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testDualWrite(ClusterInstance) > [2] Type=ZK, MetadataVersion=3.5-IV2, Security=PLAINTEXT STARTED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testDualWrite(ClusterInstance) > [2] Type=ZK, MetadataVersion=3.5-IV2, Security=PLAINTEXT PASSED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testDualWrite(ClusterInstance) > [3] Type=ZK, MetadataVersion=3.6-IV2, Security=PLAINTEXT STARTED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testDualWrite(ClusterInstance) > [3] Type=ZK, MetadataVersion=3.6-IV2, Security=PLAINTEXT PASSED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testDualWrite(ClusterInstance) > [4] Type=ZK, MetadataVersion=3.7-IV0, Security=PLAINTEXT STARTED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testDualWrite(ClusterInstance) > [4] Type=ZK, MetadataVersion=3.7-IV0, Security=PLAINTEXT PASSED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testDualWrite(ClusterInstance) > [5] Type=ZK, MetadataVersion=3.7-IV1, Security=PLAINTEXT STARTED

Gradle Test Run :core:test > Gradle Test Executor 1 > ZkMigrationIntegrationTest > testDualWrite(ClusterInstance) > [5] Type=ZK, MetadataVersion=3.7-IV1, Security=PLAINTEXT PASSED

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.3/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 2m
55 actionable tasks: 1 executed, 54 up-to-date

See the profiling report at: file:///home/ubuntu/kafka/build/reports/profile/profile-2023-11-16-19-03-58.html
A fine-grained performance profile is available: use the --scan option.

Figured out the exact cause for failures, so the shared power nodes don't have Scala installed. Will raise a JIRA ticket and get SCALA 2.13.12 installed

Nov 16 '23 19:11 Vaibhav-Nazare

JIRA ticket for Scala Installation on shared power nodes: https://issues.apache.org/jira/browse/INFRA-25163

Nov 16 '23 19:11 Vaibhav-Nazare

Hi @mimaison any updates on the latest changes?

Nov 30 '23 09:11 Vaibhav-Nazare

Hi Team , any updates on the plan to merge this PR?

Dec 27 '23 05:12 Vaibhav-Nazare

Sorry for the delay. I tried to build this but it fails in the CI with the following error:

WorkflowScript: 91: unexpected token: % @ line 91, column 82.
   InMillis() / 1000 / 60 / 60 / % 24

See https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14717/6/pipeline

Jan 04 '24 16:01 mimaison

kafka kafka copied to clipboard

KAFKA-15062: Adding ppc64le build stage

Committer Checklist (excluded from commit message)

kafka
kafka copied to clipboard