celeborn
celeborn copied to clipboard
[CELEBORN-1617][CIP-11] Support workers tags in FS Config Service
What changes were proposed in this pull request?
Adding support for reading worker tags in FSConfigService. (Will handle DBConfigService in separate PR). Tags will be part of System level config and dynamic config file structure with tags is shown below:
- level: SYSTEM
config:
celeborn.test.int.only: 100
tags:
tag1:
- 'host1:1111'
- 'host2:2222'
tag2:
- 'host3:3333'
- 'host4:4444'
Why are the changes needed?
https://cwiki.apache.org/confluence/display/CELEBORN/CIP-11+Supporting+Tags+in+Celeborn
Does this PR introduce any user-facing change?
- Changes are backward compatible.
- User will be able to pass worker tags from dynamicConfig.yaml file.
How was this patch tested?
UTs
I've made tags part of SystemConfig as it made more sense to me, let me know if you think if it should extracted out as a separate config something like below.
- tags:
tag1:
- 'host1:1111'
- 'host2:2222'
tag2:
- 'host3:3333'
- 'host4:4444'
cc: @FMX @zwangsheng @mridulm
ping @FMX @zwangsheng @mridulm