Feature request: Provide an option to hide "system" topics

[AlexeyRaga]

Topics that are starting with __ are often considered "system" in Kafka Ecosystem.

It would be nice to have an option (a checkbox) to allow showing/hiding "system" topics.

[weeco]

@rikimaru0345 It seems like we lost the "Hide internal topics" toggle at some point.

[cristi-maco]

Question: will/should the hiding apply also to topic name starting with only 1 underscore, not only 2?
Asking because some other UI client has a similar option which hides both types of names, and I find it useful, as I have some topics with names configured by me, but which I still consider kind of internal and would prefer to be hidden (like some topics needed by Kafka Connect), so I use 1 underscore prefix for them ("_connect-offsets"), unlike other internal ones whose names I don't control ("__consumer-offsets")

[weeco]

Technically the name does not indicate whether it's an internal topic or not (the Kafka protocol has an extra property for internal topicsi) - it's just a convention. So I think we should not be opinionated about this and hide single underscore prefix as other users may not expect them to be hidden as an internal topic.

I believe hiding all topics with double underscores makes the most sense as of today, but we could make this configurable nonetheless.

[michaelpearce-gain]

in lenses this is configurable, where it can be fully qualified or wildcarded topic name, it would be good to have similar as well as a toggle to be able to still see those system topics, just by default the toggle is off to limit noise.

e.g. this is our lenses.io config where we are able to hide these topics by default (but in ui can toggle them back in view if needed)

  #Extend the control (system/internal) topics for "_*" and "__*" and add default list here: https://docs.lenses.io/4.3/configuration/options/basic/#control-topics 
  LENSES_KAFKA_CONTROL_TOPICS: | 
    [
      "_schemas",
      "__consumer_offsets",
      "_kafka_lenses_",
      "lsql_",
      "lsql-",
      "__transaction_state",
      "__topology",
      "__topology__metrics",
      "*connect_configs*",
      "*connect_status*",
      "*connect_offsets*",
      "*connect_statuses*",
      "_confluent*",
      "*-KSTREAM-*",
      "*-TableSource-*",
      "*-changelog",
      "__amazon_msk*",
      "__*",
      "_*"
    ]

[weeco]

I'm closing this because we added support for this. Currently the filter logic only includes topics with a double prefix. We may have another look at this, thanks for providing the list of potential system topics @michaelpearce-gain

[michaelpearce-gain]

If you are closing this, can you open a new one, to track ability for users to customise this list and support wildcards / pattern matching

[michaelpearce-gain]

@weeco did you create a breakout to capture the bits not done yet?