starrocks icon indicating copy to clipboard operation
starrocks copied to clipboard

[Feature]Use jprotobuf-precompile-plugin for starrocks

Open Lrabbit125 opened this issue 1 year ago • 3 comments

Why I'm doing:

The jprotobuf code generation plug-in used by StarRocks provides a pre-compiled alternative, eliminating the need for runtime code generation via the JDK’s compilation API. This improvement enhances the efficiency of RPC information transmission.

What I'm doing:

Add the precompile plugin to the pom file and modify the current compilation check.

What type of PR is this:

  • [ ] BugFix
  • [x] Feature
  • [ ] Enhancement
  • [ ] Refactor
  • [ ] UT
  • [ ] Doc
  • [ ] Tool

Does this PR entail a change in behavior?

  • [x] Yes, this PR will result in a change in behavior.
  • [ ] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • [ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • [ ] Parameter changes: default values, similar parameters but with different default values
  • [x] Policy changes: use new policy to replace old one, functionality automatically enabled
  • [ ] Feature removed
  • [ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • [ ] I have added test cases for my bug fix or my new feature
  • [ ] This pr needs user documentation (for new or modified features or behaviors)
    • [ ] I have added documentation for my new feature or new function
  • [x] This is a backport pr

Bugfix cherry-pick branch check:

  • [ ] I have checked the version labels which the pr will be auto-backported to the target branch
    • [ ] 3.3
    • [ ] 3.2
    • [ ] 3.1
    • [ ] 3.0
    • [ ] 2.5

Lrabbit125 avatar Aug 26 '24 13:08 Lrabbit125

Please open a github issue to associate to this [Feature] development.

Sorry, I accidentally closed this PR. It is now associated with the issue #50684.

Lrabbit125 avatar Sep 04 '24 09:09 Lrabbit125

introduces new CVE

==========
Total: 5 (HIGH: 3, CRITICAL: 2)

┌────────────────────────────────────────────────────────────┬──────────────────┬──────────┬────────┬───────────────────┬────────────────────────────────┬─────────────────────────────────────────────────────────────┐
│                          Library                           │  Vulnerability   │ Severity │ Status │ Installed Version │         Fixed Version          │                            Title                            │
├────────────────────────────────────────────────────────────┼──────────────────┼──────────┼────────┼───────────────────┼────────────────────────────────┼─────────────────────────────────────────────────────────────┤
│ com.google.protobuf:protobuf-java                          │ CVE-2022-3509    │ HIGH     │ fixed  │ 3.17.3            │ 3.16.3, 3.19.6, 3.20.3, 3.21.7 │ protobuf-java: Textformat parsing issue leads to DoS        │
│ (orc-mapreduce-1.8.7-shaded-protobuf.jar)                  │                  │          │        │                   │                                │ https://avd.aquasec.com/nvd/cve-2022-3509                   │
│                                                            ├──────────────────┤          │        │                   │                                ├─────────────────────────────────────────────────────────────┤
│                                                            │ CVE-2022-3510    │          │        │                   │                                │ protobuf-java: Message-Type Extensions parsing issue leads  │
│                                                            │                  │          │        │                   │                                │ to DoS                                                      │
│                                                            │                  │          │        │                   │                                │ https://avd.aquasec.com/nvd/cve-2022-3510                   │
├────────────────────────────────────────────────────────────┼──────────────────┼──────────┤        ├───────────────────┼────────────────────────────────┼─────────────────────────────────────────────────────────────┤
│ org.apache.maven:maven-core (maven-core-2.0.9.jar)         │ CVE-2021-26291   │ CRITICAL │        │ 2.0.9             │ 3.8.1                          │ maven: Block repositories using http by default             │
│                                                            │                  │          │        │                   │                                │ https://avd.aquasec.com/nvd/cve-2021-26291                  │
├────────────────────────────────────────────────────────────┼──────────────────┤          │        ├───────────────────┼────────────────────────────────┼─────────────────────────────────────────────────────────────┤
│ org.codehaus.plexus:plexus-utils (plexus-utils-1.5.15.jar) │ CVE-2017-1000487 │          │        │ 1.5.15            │ 3.0.16                         │ plexus-utils: Mishandled strings in Commandline class allow │
│                                                            │                  │          │        │                   │                                │ for command injection                                       │
│                                                            │                  │          │        │                   │                                │ https://avd.aquasec.com/nvd/cve-2017-1000487                │
│                                                            ├──────────────────┼──────────┤        │                   ├────────────────────────────────┼─────────────────────────────────────────────────────────────┤
│                                                            │ CVE-2022-4244    │ HIGH     │        │                   │ 3.0.24                         │ codehaus-plexus: Directory Traversal                        │
│                                                            │                  │          │        │                   │                                │ https://avd.aquasec.com/nvd/cve-2022-4244                   │
└────────────────────────────────────────────────────────────┴──────────────────┴──────────┴────────┴───────────────────┴────────────────────────────────┴─────────────────────────────────────────────────────────────┘

kevincai avatar Sep 05 '24 09:09 kevincai

the PR is great, need to fix the introduced CVEs.

The CVE issues have already been resolved.

Lrabbit125 avatar Sep 07 '24 07:09 Lrabbit125

@mergify rebase

nshangyiming avatar Sep 07 '24 09:09 nshangyiming

rebase

✅ Branch has been successfully rebased

mergify[bot] avatar Sep 07 '24 09:09 mergify[bot]

@mergifyio rebase

kevincai avatar Sep 09 '24 01:09 kevincai

rebase

✅ Branch has been successfully rebased

mergify[bot] avatar Sep 09 '24 01:09 mergify[bot]

[Java-Extensions Incremental Coverage Report]

:white_check_mark: pass : 0 / 0 (0%)

github-actions[bot] avatar Sep 09 '24 04:09 github-actions[bot]

[FE Incremental Coverage Report]

:white_check_mark: pass : 0 / 0 (0%)

github-actions[bot] avatar Sep 09 '24 04:09 github-actions[bot]

[BE Incremental Coverage Report]

:white_check_mark: pass : 0 / 0 (0%)

github-actions[bot] avatar Sep 09 '24 04:09 github-actions[bot]

@Lrabbit125 this plugin requires some packages can not be in scope of provided. If I want to like comment hudi-coommon or paimon-bundle to provided scope, how can I do that?

dirtysalt avatar Mar 08 '25 19:03 dirtysalt

@Lrabbit125 BTW, can you elaborate what this plugin is for? or share me some links about principles of this enhancement?

dirtysalt avatar Mar 08 '25 19:03 dirtysalt

And if I mark paimon-bundle as scope provided, it fails like

java.lang.reflect.InvocationTargetException
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:568)
    at com.baidu.jprotobuf.mojo.PreCompileMojo$1.run (PreCompileMojo.java:326)
    at java.lang.Thread.run (Thread.java:842)
Caused by: jodd.io.findfile.FindFileException: Scan entry error: EntryData{com.starrocks.planner.PaimonScanNode'}; <--- java.lang.ClassNotFoundException: org.apache.paimon.table.source.Split
    at jodd.io.findfile.ClassFinder.scanEntry (ClassFinder.java:377)
    at jodd.io.findfile.ClassFinder.scanClassFile (ClassFinder.java:324)
    at jodd.io.findfile.ClassFinder.scanClassPath (ClassFinder.java:307)
    at jodd.io.findfile.ClassFinder.scanPath (ClassFinder.java:239)
    at jodd.io.findfile.ClassFinder.scanPaths (ClassFinder.java:178)
    at jodd.io.findfile.ClassScanner.scan (ClassScanner.java:35)
    at jodd.io.findfile.ClassScanner.scanDefaultClasspath (ClassScanner.java:28)
    at com.baidu.jprotobuf.mojo.JprotobufPreCompileMain.main (JprotobufPreCompileMain.java:182)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:568)
    at com.baidu.jprotobuf.mojo.PreCompileMojo$1.run (PreCompileMojo.java:326)
    at java.lang.Thread.run (Thread.java:842)
Caused by: java.lang.Exception: org/apache/paimon/table/source/Split
    at com.baidu.jprotobuf.mojo.JprotobufPreCompileMain$1.onEntry (JprotobufPreCompileMain.java:177)
    at jodd.io.findfile.ClassFinder.scanEntry (ClassFinder.java:375)
    at jodd.io.findfile.ClassFinder.scanClassFile (ClassFinder.java:324)
    at jodd.io.findfile.ClassFinder.scanClassPath (ClassFinder.java:307)
    at jodd.io.findfile.ClassFinder.scanPath (ClassFinder.java:239)
    at jodd.io.findfile.ClassFinder.scanPaths (ClassFinder.java:178)
    at jodd.io.findfile.ClassScanner.scan (ClassScanner.java:35)
    at jodd.io.findfile.ClassScanner.scanDefaultClasspath (ClassScanner.java:28)
    at com.baidu.jprotobuf.mojo.JprotobufPreCompileMain.main (JprotobufPreCompileMain.java:182)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:568)
    at com.baidu.jprotobuf.mojo.PreCompileMojo$1.run (PreCompileMojo.java:326)
    at java.lang.Thread.run (Thread.java:842)

dirtysalt avatar Mar 08 '25 19:03 dirtysalt

@Lrabbit125 If I understand correctly, the main purpose of this pr is to let starrocks run in JRE mode (instead of JDK mode).

And I try my fix https://github.com/StarRocks/starrocks/pull/56732 on JRE17(zulu), and it works.

dirtysalt avatar Mar 10 '25 19:03 dirtysalt