databend icon indicating copy to clipboard operation
databend copied to clipboard

Published 20 hours ago • verified publisher icon datafuselabs

Refactor: upgrade SettingMgr to using protobuf

Open drmingdrmer opened this issue 11 months ago • 2 comments

The problem

Currently SettingMgr write json data into meta-service: https://github.com/drmingdrmer/databend/blob/38e20916335824f70c7c553395fd380efd77603c/src/query/management/src/setting/setting_mgr.rs#L73

Json does not provide compatibility support as FromToProto does: https://github.com/drmingdrmer/databend/blob/6332061fb2c8466c398b417cc6db878ba5933740/src/meta/proto-conv/src/from_to_protobuf.rs#L18

The setting data read/write should be upgrade to using FromToProto to provide well defined and tested backward compatibility.

TODO

  • [ ] keep the writing part unchanged, but read in pb/json compatible way.
  • [ ] upgrade to write pb data instead of json.
  • [ ] upgrade to read only pb format.

drmingdrmer avatar Mar 19 '24 04:03 drmingdrmer

Hi @drmingdrmer,

I'm interested in working on this issue. if it's still available for assignment. Could you please assign it to me?

shamb0 avatar May 09 '24 11:05 shamb0

Thank you, but we have our own plan for migrating the format.

This job likely involves other related components in our code base, as it is not very standalone.

The first step involves upgrading our system to read and load both JSON and the designated message protocol. The second step will be to rewrite the writing process so that it only records properly formatted messages into the meta service.

drmingdrmer avatar May 09 '24 12:05 drmingdrmer