gobblin icon indicating copy to clipboard operation
gobblin copied to clipboard

GOBBLIN-1055 Gaas jobs not able to write to state store

Open chris9692 opened this issue 5 years ago • 1 comments

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

  • [X] My PR addresses the following [Gobblin JIRA]
    • https://issues.apache.org/jira/browse/GOBBLIN-1055

Description

  • [X] Here are some details about my PR, including screenshots (if applicable):

The change pass the GobblinClassLoader to Hadoop through Configuration so that Hadoop can load the serializer properly.

Tests

  • [X] My PR adds the following unit tests OR does not need testing for this extremely good reason:

PR fixed a loop hole in passing parameters from Gobblin-metastore to Hadoop, and it was tested effective in Gaas flow execution with state.store.enabled=true.

Commits

  • [X] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

chris9692 avatar Feb 19 '20 04:02 chris9692

Codecov Report

Merging #2895 into master will increase coverage by <.01%. The diff coverage is 100%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master    #2895      +/-   ##
============================================
+ Coverage     45.82%   45.82%   +<.01%     
+ Complexity     9151     9149       -2     
============================================
  Files          1929     1929              
  Lines         72625    72626       +1     
  Branches       7995     7995              
============================================
+ Hits          33279    33281       +2     
+ Misses        36309    36308       -1     
  Partials       3037     3037
Impacted Files Coverage Δ Complexity Δ
...ava/org/apache/gobblin/metastore/FsStateStore.java 36.48% <100%> (+0.43%) 14 <0> (ø) :arrow_down:
...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java 63.33% <0%> (-1.12%) 15% <0%> (-1%)
.../org/apache/gobblin/cluster/GobblinTaskRunner.java 64.62% <0%> (-0.44%) 28% <0%> (-1%)
...lin/restli/throttling/ZookeeperLeaderElection.java 72.22% <0%> (+2.22%) 13% <0%> (ø) :arrow_down:
...lin/util/filesystem/FileSystemInstrumentation.java 92.85% <0%> (+7.14%) 3% <0%> (ø) :arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 2991f42...540c734. Read the comment docs.

codecov-io avatar Feb 19 '20 04:02 codecov-io