[GOBBLIN-2136] Byebye dag manager

[arjun4084346]

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

• [x] My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
  • https://issues.apache.org/jira/browse/GOBBLIN-2136

Description

• [x] Here are some details about my PR, including screenshots (if applicable):

Now when we have developed the new GaaS architecture based on DagProcessingEngine, we do not need DagManager and related classes. This also enabled GaaS to work in a multi leader mode, so there is no need for the code that was there for standby hosts.

a) Removed the code path for GOBBLIN_SERVICE_FLOW_CATALOG_LOCAL_COMMIT = false Removed the code path for FORCE_LEADER = true Removed the code path for MULTI_ACTIVE_EXECUTION_ENABLED = false Removed the code path for MULTI_ACTIVE_SCHEDULER_ENABLED = false Removed the code path for GOBBLIN_SERVICE_WARM_STANDBY_ENABLED_KEY = false Removed the code path for DAG_PROC_ENGINE_ENABLED = false

b) Merged FlowConfigsResourceHandler, FlowConfigV2ResourceLocalHandler, FlowConfigResourceLocalHandler, FlowConfigsV2ResourceHandler, GobblinServiceFlowConfigResourceHandler, GobblinServiceFlowConfigV2ResourceHandler, GobblinServiceFlowConfigV2ResourceHandlerWithWarmStandby into FlowConfigsV2ResourceHandler

Merged FlowExecutionResourceLocalHandler, GobblinServiceFlowExecutionResourceHandler, GobblinServiceFlowExecutionResourceHandlerWithWarmStandby into FlowExecutionResourceHandler

Merged FlowConfigClient into FlowConfigV2Client along with their test classes

Merged DagManagementDagActionStoreChangeMonitor into DagActionStoreChangeMonitor along with their test classes

c) Removed DagManager, Helix and EventBus related code from GaaS

d) Removed Optional from DagManagement, DagTaskStream, DagProcFactory, DagProcessingEngine, DagManagementStateStore, FlowLaunchHandler, DagActionReminderScheduler

e) Codestyle changes suggested by IntelliJ

Tests

• [x] My PR adds the following unit tests OR does not need testing for this extremely good reason: no code logic change, only updated the existing tests

Commits

• [x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 68.04124% with 93 lines in your changes missing coverage. Please review.

Project coverage is 48.25%. Comparing base (444f266) to head (349f2b9). Report is 4 commits behind head on master.

Files with missing lines	Patch %	Lines
.../modules/scheduler/GobblinServiceJobScheduler.java	55.81%	19 Missing :warning:
...les/orchestration/DagManagementTaskStreamImpl.java	10.52%	17 Missing :warning:
...e/modules/orchestration/MysqlUserQuotaManager.java	0.00%	14 Missing :warning:
...rvice/modules/orchestration/DagManagerMetrics.java	44.44%	5 Missing :warning:
...in/service/modules/core/GobblinServiceManager.java	71.42%	4 Missing :warning:
...odules/orchestration/InMemoryUserQuotaManager.java	78.94%	2 Missing and 2 partials :warning:
...in/service/modules/orchestration/Orchestrator.java	69.23%	2 Missing and 2 partials :warning:
...rvice/monitoring/KafkaJobStatusMonitorFactory.java	0.00%	4 Missing :warning:
...ache/gobblin/runtime/spec_catalog/FlowCatalog.java	62.50%	2 Missing and 1 partial :warning:
...obblin/service/modules/orchestration/DagUtils.java	62.50%	3 Missing :warning:
... and 11 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #4031      +/-   ##
============================================
+ Coverage     38.79%   48.25%   +9.45%     
- Complexity     1599     9111    +7512     
============================================
  Files           388     1778    +1390     
  Lines         15998    68211   +52213     
  Branches       1585     7315    +5730     
============================================
+ Hits           6207    32913   +26706     
- Misses         9293    32513   +23220     
- Partials        498     2785    +2287     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.