gobblin icon indicating copy to clipboard operation
gobblin copied to clipboard

[Gobblin-982] Fix JDBC extraction dependencies issue

Open PnPie opened this issue 5 years ago • 5 comments

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

  • [x] My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-982] Dependencies missing for running JDBC extraction job"
    • https://issues.apache.org/jira/browse/GOBBLIN-982

Description

  • [x] Here are some details about my PR, including screenshots (if applicable):

Add "gobblin-sql" module into distribution dependencies so that it could be packaged into built tarball, otherwise while running JDBC extraction job, it will pop up ClassNotFound exception for all "gobblin-sql" classes.

Add jdbc related libraries into mr libs as jdbc extraction codes are actually running on mappers and need these dependencies (same as above got ClassNotFound exception otherwise).

Tests

  • [x] My PR adds the following unit tests OR does not need testing for this extremely good reason: Gradle config and shell scripts modification.

Commits

  • [x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

PnPie avatar Nov 26 '19 14:11 PnPie

@PnPie Thanks for fixing this, I had to fix the same, this looks good but can u pls take a look at the jenkins failure? If it looks intermittent, pls just push this again to trigger the CI. Thanks

jhsenjaliya avatar Mar 19 '20 21:03 jhsenjaliya

Hi Jay @jhsenjaliya ! CI is in maintenance and I merged master in to trigger it again.

PnPie avatar Mar 21 '20 09:03 PnPie

looks like the build is getting stalled, I also see many elasticsearch exceptions, not sure if that is causing any test failure though.

jhsenjaliya avatar Mar 22 '20 02:03 jhsenjaliya

Yes seems the elasticsearch exceptions are on the internal ES test server, and at the moment will not impact the build, I can successfully build the ES module locally, so I just pushed some logging info changes to trigger it again.

PnPie avatar Mar 22 '20 06:03 PnPie

Codecov Report

Merging #2828 into master will increase coverage by 0.95%. The diff coverage is 0.00%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master    #2828      +/-   ##
============================================
+ Coverage     44.60%   45.55%   +0.95%     
- Complexity     8980     9149     +169     
============================================
  Files          1936     1936              
  Lines         73234    73234              
  Branches       8083     8083              
============================================
+ Hits          32669    33365     +696     
+ Misses        37515    36773     -742     
- Partials       3050     3096      +46     
Impacted Files Coverage Δ Complexity Δ
.../org/apache/gobblin/source/jdbc/JdbcExtractor.java 9.40% <0.00%> (ø) 19.00 <0.00> (ø)
...lin/util/filesystem/FileSystemInstrumentation.java 85.71% <0.00%> (-7.15%) 3.00% <0.00%> (ø%)
...ava/org/apache/gobblin/fsm/FiniteStateMachine.java 73.48% <0.00%> (-3.04%) 18.00% <0.00%> (-3.00%)
...lin/restli/throttling/ZookeeperLeaderElection.java 70.00% <0.00%> (-2.23%) 13.00% <0.00%> (ø%)
.../org/apache/gobblin/metrics/RootMetricContext.java 78.12% <0.00%> (-1.57%) 15.00% <0.00%> (-1.00%)
...src/main/java/org/apache/gobblin/runtime/Task.java 67.26% <0.00%> (+0.22%) 84.00% <0.00%> (+1.00%)
.../org/apache/gobblin/runtime/SafeDatasetCommit.java 47.31% <0.00%> (+0.48%) 29.00% <0.00%> (ø%)
...rg/apache/gobblin/runtime/AbstractJobLauncher.java 60.36% <0.00%> (+0.77%) 37.00% <0.00%> (+2.00%)
...ain/java/org/apache/gobblin/runtime/fork/Fork.java 75.00% <0.00%> (+0.83%) 68.00% <0.00%> (ø%)
.../apache/gobblin/runtime/api/JobExecutionState.java 80.37% <0.00%> (+0.93%) 24.00% <0.00%> (ø%)
... and 41 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update a634612...ab44e56. Read the comment docs.

codecov-io avatar Mar 22 '20 07:03 codecov-io