[GOBBLIN-939] Integrate usage of env variables in gobblin scripts and standardize configs

[jhsenjaliya]

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

• [x] My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
  • https://issues.apache.org/jira/browse/GOBBLIN-939

Description

• [x] Here are some details about my PR, including screenshots (if applicable):

1. standardize config with ENV variables
2. define default env variables in gobblin-env.sh

Tests

• [x] My PR adds the following unit tests OR does not need testing for this extremely good reason: N/A

Commits

• [x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

[Will-Lo]

I was wondering, would it be better if we were to utilise javaopts, so -D flags for these environment variables, rather than continuously modifying the scripts/configs?

[codecov-io]

Codecov Report

Merging #2788 (442d421) into master (f3d75e3) will decrease coverage by 0.94%. The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #2788      +/-   ##
============================================
- Coverage     45.87%   44.93%   -0.95%     
+ Complexity     9190     9019     -171     
============================================
  Files          1934     1934              
  Lines         72931    72931              
  Branches       8046     8046              
============================================
- Hits          33458    32771     -687     
- Misses        36404    37132     +728     
+ Partials       3069     3028      -41     

Impacted Files	Coverage Δ	Complexity Δ
...gobblin/runtime/mapreduce/GobblinOutputFormat.java	0.00% <0.00%> (-100.00%)	0.00% <0.00%> (-2.00%)
...askStateCollectorServiceHiveRegHandlerFactory.java	0.00% <0.00%> (-100.00%)	0.00% <0.00%> (-2.00%)
...re/filesystem/FsDatasetStateStoreEntryManager.java	0.00% <0.00%> (-100.00%)	0.00% <0.00%> (-3.00%)
...in/runtime/mapreduce/CustomizedProgresserBase.java	0.00% <0.00%> (-83.34%)	0.00% <0.00%> (-1.00%)
...rg/apache/gobblin/runtime/ZkDatasetStateStore.java	0.00% <0.00%> (-80.77%)	0.00% <0.00%> (-7.00%)
...lin/runtime/locks/LegacyJobLockFactoryManager.java	0.00% <0.00%> (-78.58%)	0.00% <0.00%> (-2.00%)
.../apache/gobblin/metastore/ZkStateStoreFactory.java	0.00% <0.00%> (-71.43%)	0.00% <0.00%> (-2.00%)
...he/gobblin/runtime/ZkDatasetStateStoreFactory.java	0.00% <0.00%> (-71.43%)	0.00% <0.00%> (-2.00%)
...apache/gobblin/runtime/MysqlDatasetStateStore.java	0.00% <0.00%> (-70.00%)	0.00% <0.00%> (-7.00%)
...e/HiveRegTaskStateCollectorServiceHandlerImpl.java	0.00% <0.00%> (-69.24%)	0.00% <0.00%> (-3.00%)
... and 38 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update f3d75e3...c5d8b5f. Read the comment docs.

[jhsenjaliya]

I was wondering, would it be better if we were to utilise javaopts, so -D flags for these environment variables, rather than continuously modifying the scripts/configs?

u mean like javaopts in bash ?

[jhsenjaliya]

@Will-Lo , @sv2000 can u pls take a look, This will make all exec mode config very standard with the gobblin script changes. Thanks

[jhsenjaliya]

@Will-Lo,can u pls review this ?

[Will-Lo]

I'll take a look, thanks!

[Will-Lo]

Going to test this out first later today, thanks for the additions!

[jhsenjaliya]

fixed the typo.

[jhsenjaliya]

@Will-Lo , @sv2000 , Can u pls take a look?