redpanda icon indicating copy to clipboard operation
redpanda copied to clipboard

Feature: offline_log_viewer updates

Open andijcr opened this issue 3 years ago • 3 comments
trafficstars

Cover letter

These changes add a first support in tools/offlile_log_viewer for serde-encoded controller messages.

running

python viewer.py --type controller --path [redpanda data path]

will decode logs produced by v22.2 and earlier. serde commands start with a field 'envelope' that contains [version, compat_version, encoded_size].

Partially fixes #5293,

implements:

  • [x] serde decode of controller commands:
    • [x] decode_topic_command
    • [x] decode_user_command
    • [x] decode_acl_command
    • [x] decode_config_command
    • [x] decode_feature_command
    • [x] decode_node_management
  • [ ] gold master and coverage testing
  • [ ] ductape integration
    • [ ] test that new version of redpanda can be completely decoded with this tool
    • [ ] add means to run this tool as a part of a ductape test

open questions:

  • should there be a check (with a message) that all the bytes are processed?
  • should the structure of the json reflect more closely the cpp class hierarchies (some hierarchies are flattened)?
  • should the names of the fields match the one used in the cpp source (some names are abbreviated or de-abbreviated)?

Backport Required

  • [ ] not a bug fix
  • [ ] issue does not exist in previous branches
  • [x] papercut/not impactful enough to backport
  • [ ] v22.2.x
  • [ ] v22.1.x
  • [ ] v21.11.x

UX changes

messages are decoded in a stream of json files. the schema for adl encoded files is unchanged, but the schema for serde - encoded message is different in some places than the corresponding adl-one, reflecting more the hierarchical data organization.

Release notes

  • none

andijcr avatar Aug 25 '22 16:08 andijcr

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Aug 25 '22 16:08 CLAassistant

@andijcr can you merge https://github.com/redpanda-data/redpanda/pull/5315 into this branch as a starting point for tests? I just rebased that PR but I expect it to fail without the changes in this PR to add serde support.

jcsp avatar Sep 20 '22 14:09 jcsp

(didn't review the code but I notice the commit messages aren't in the usual format: see CONTRIBUTING.md in the source and/or take a peek at the commit history)

jcsp avatar Sep 20 '22 14:09 jcsp

@andijcr this has been stalled for a while -- it's not our top priority, but it would be good to know if there were issues blocking this

jcsp avatar Nov 24 '22 19:11 jcsp

I've merged this into https://github.com/redpanda-data/redpanda/pull/5315

jcsp avatar Dec 02 '22 14:12 jcsp