[ZEPPELIN-5815] Frontend CI jobs improvement

[huage1994]

trafficstars

What is this PR for?

This PR is to improve a some step in Frontend CI jobs.

There are two command in this step in file frontend.yml as follow: https://github.com/apache/zeppelin/blob/f42943c6d5bc04498b7fc9ca518dc433091a9c9c/.github/workflows/frontend.yml#L123-L125

The first command would mvn install many modules including the submodule of zeppelin-plugins. The second command would mvn package zeppelin-plugins.

Actually, we only need to mvn package module zeppelin-plugins one time instead of to install then to package it, which would save running time of CI. And there is no need to mvn install module zeppelin-plugins , which may bring something unexpected. mvn package is enough.

What type of PR is it?

Improvement

Todos

• [ ] - Task

What is the Jira issue?

• Open an issue on Jira ZEPPELIN-5815

How should this be tested?

CI passed

Screenshots (if appropriate)

Questions:

• Does the licenses files need to update? No
• Is there breaking changes for older versions? No
• Does this needs documentation? No

[Reamer]

Currently, the compilation of the plugins is quite fast. Where did you notice that it takes a long time? Source: https://github.com/apache/zeppelin/actions/runs/3059081927/jobs/4936042114

[INFO] Zeppelin: Plugins Parent ........................... SUCCESS [  4.729 s]
[INFO] Zeppelin: Plugin S3NotebookRepo .................... SUCCESS [  5.785 s]
[INFO] Zeppelin: Plugin GitHubNotebookRepo ................ SUCCESS [  5.088 s]
[INFO] Zeppelin: Plugin AzureNotebookRepo ................. SUCCESS [  4.792 s]
[INFO] Zeppelin: Plugin GCSNotebookRepo ................... SUCCESS [  5.270 s]
[INFO] Zeppelin: Plugin FileSystemNotebookRepo ............ SUCCESS [  5.207 s]
[INFO] Zeppelin: Plugin MongoNotebookRepo ................. SUCCESS [  5.177 s]
[INFO] Zeppelin: Plugin OSSNotebookRepo ................... SUCCESS [  5.037 s]
[INFO] Zeppelin: Plugin Kubernetes StandardLauncher ....... SUCCESS [  9.391 s]
[INFO] Zeppelin: Plugin Flink Launcher .................... SUCCESS [  6.069 s]
[INFO] Zeppelin: Plugin Docker Launcher ................... SUCCESS [  5.728 s]
[INFO] Zeppelin: Plugin Cluster Launcher .................. SUCCESS [  5.420 s]
[INFO] Zeppelin: Plugin Yarn Launcher ..................... SUCCESS [  5.052 s]

[huage1994]

Currently, the compilation of the plugins is quite fast. Where did you notice that it takes a long time? Source:

Hi @Reamer, I didn't mean mvn package zeppelin-plugins would take much time.
Currently, front end CI would firstly mvn install zeppelin-plugins and then mvn package zeppelin-plugins. This is not necessary. This PR is to remove mvn install zeppelin-plugins.

Currently there are two command in this step in file frontend.yml as follow: https://github.com/apache/zeppelin/blob/f42943c6d5bc04498b7fc9ca518dc433091a9c9c/.github/workflows/frontend.yml#L123-L125

The first command would mvn install many modules including the submodule of zeppelin-plugins.

[huage1994]

Hi @Reamer, I'm sorry that I didn't make it clear. Now I have updated the description and my comments. I think the following description is kind of misleading. Actually it shouldn't be put here. I have removed it from the description and make it into a comment of #4321 .

By the way when I upgrade junit 4 to junit 5 https://github.com/apache/zeppelin/pull/4321 , Frontend CI job would block at downloading dependency when mvn install the module k8s-standard, but it is ok to mvn package this module. It 's really weird that both mvn install and mvn package is OK in my local environment, this problem only happens in Github CI jobs . It has costed me very much time, but hard to find the real reason.I would really appreciate if someone tell me the reason.

Removing mvn install zeppelin-plugins would also help to solve the problem mentioned the comment in #4321 .

[zjffdu]

Frontend CI is passed，Thanks @huage1994 LGTM

[Reamer]

In general, I am against additional Maven calls. Maybe it makes sense to work with the Maven parameters "-am" and "-pl" in the first call. This should build only the most necessary artefacts, which have been specified with "pl". See: https://maven.apache.org/guides/mini/guide-multiple-modules-4.html#command-line-options

[huage1994]

In general, I am against additional Maven calls. Maybe it makes sense to work with the Maven parameters "-am" and "-pl" in the first call. This should build only the most necessary artefacts, which have been specified with "pl". See: https://maven.apache.org/guides/mini/guide-multiple-modules-4.html#command-line-options

Hi @Reamer , I don't know if I get your point.

Zeppelin uses maven 3.8.1 . mvn -pl parent-module would ignores any child modules.

If we only use "-pl" instead of -DskipZeppelinPlugins here, we have to specify all 11 children modules of zeppelin-plugins like this: !zeppelin-plugins/launcher/cluster//,!zeppelin-plugins/launer/docker,!zeppelin-plugins/notebookrepo/azure,... That's the reason I add -DskipZeppelinPlugins here.

[huage1994]

Zeppelin uses maven 3.8.1 . mvn -pl parent-module would ignores any child modules.

/mvnw clean package -pl zeppelin-plugins -amd -B Here -amd is a trick to run maven command recursively.

[Reamer]

Sorry if I was vague here. Let me describe it in a little more detail. During the tests the project zeppelin-integration is executed. In my opinion this sub-module has to define all dependencies in the pom.xml so that the tests can be executed successfully, without install the whole zeppelin project via mvn install before. Unfortunately this is not yet the case, because only a few dependencies are needed during the installation.

./mvnw package -pl zeppelin-integration -Pintegration -am -DskipTests
[INFO] Scanning for projects...
[WARNING] The project org.apache.zeppelin:zeppelin-integration:jar:0.11.0-SNAPSHOT uses prerequisites which is only intended for maven-plugin projects but not for non maven-plugin projects. For such purposes you should use the maven-enforcer-plugin. See https://maven.apache.org/enforcer/enforcer-rules/requireMavenVersion.html
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] Zeppelin                                                           [pom]
[INFO] Zeppelin: Common                                                   [jar]
[INFO] Zeppelin: Interpreter                                              [jar]
[INFO] Zeppelin: Jupyter Support                                          [jar]
[INFO] Zeppelin: Zengine                                                  [jar]
[INFO] Zeppelin: Integration Test                                         [jar]

My point is that it may not be necessary to install the Zeppelin plugins at all.

[huage1994]

Hi @Reamer @jongyoul @zjffdu , I'm sorry I deleted this branch by mistake，I would submit a new PR