celeborn icon indicating copy to clipboard operation
celeborn copied to clipboard

[CELEBORN-1617][CIP-11] Support workers tags in FS Config Service

Open s0nskar opened this issue 1 year ago • 1 comments

What changes were proposed in this pull request?

Adding support for reading worker tags in FSConfigService. (Will handle DBConfigService in separate PR). Tags will be part of System level config and dynamic config file structure with tags is shown below:

-  level: SYSTEM
   config:
     celeborn.test.int.only: 100
   tags:
     tag1:
       - 'host1:1111'
       - 'host2:2222'
     tag2:
       - 'host3:3333'
       - 'host4:4444'

Why are the changes needed?

https://cwiki.apache.org/confluence/display/CELEBORN/CIP-11+Supporting+Tags+in+Celeborn

Does this PR introduce any user-facing change?

  • Changes are backward compatible.
  • User will be able to pass worker tags from dynamicConfig.yaml file.

How was this patch tested?

UTs

s0nskar avatar Sep 26 '24 12:09 s0nskar

I've made tags part of SystemConfig as it made more sense to me, let me know if you think if it should extracted out as a separate config something like below.

-  tags:
     tag1:
       - 'host1:1111'
       - 'host2:2222'
     tag2:
       - 'host3:3333'
       - 'host4:4444'

cc: @FMX @zwangsheng @mridulm

s0nskar avatar Sep 26 '24 12:09 s0nskar

ping @FMX @zwangsheng @mridulm

s0nskar avatar Oct 08 '24 10:10 s0nskar