Persistent query silently fails / is deleted, but still shows as RUNNING

[1riatsila1]

Describe the bug A persistent query stopped working completely. Seemingly during a maintenance window, and is in an irrecoverable state. I have not been able to reproduce the error

To Reproduce Steps to reproduce the behavior, include:

1. The version of KSQL: 7.2.1

Have a persistent query (CREATE STREAM AS SELECT in this case) Let some time pass Possibly have some broker maintenance

Expected behavior The query should operate as it always has. Or if the query enters a bad state, it should reflect as failed or we should receive a warning.

Actual behaviour A clear and concise description of what actually happens, including:

• The query stops working (data is not produced onto the target topic / stream)
• When running SHOW QUERIES I see:

<QUERY_ID>           | PERSISTENT | RUNNING:1 | <STREAM>           | <topic>           | CREATE STREAM ... AS SELECT ...

Then the following:

ksql> explain <QUERY_ID>;
Query with id:<QUERY_ID> does not exist, use SHOW QUERIES to view the full set of queries.
ksql> terminate <QUERY_ID>;
Unknown queryId: <QUERY_ID>

But running SHOW QUERIES still yeilds

<QUERY_ID>           | PERSISTENT | RUNNING:1 | <STREAM>           | <topic>           | CREATE STREAM ... AS SELECT ...

Note how the state incorrectly reflects as RUNNING

Additional context