Zhanxiang (Patrick) Huang

Please hold this PR first. Found the following panic in my env: ``` 2022-09-23T07:51:59.986194Z ERROR risingwave_storage::monitor::monitored_store: Failed in get: Hummock error: Expired Epoch: watermark 3059503322038272, epoch 3059503305588736. backtrace of inner...

> Please hold this PR first. Found the following panic in my env: > > ``` > 2022-09-23T07:51:59.986194Z ERROR risingwave_storage::monitor::monitored_store: Failed in get: Hummock error: Expired Epoch: watermark 3059503322038272, epoch...

Let's hold this PR until we figure out the cause of #4861.

Discussed offline. We think updating the metric in sstable iterator can be problematic. https://github.com/risingwavelabs/risingwave/blob/18eed8fa8543f51c244aa8c0ee4c01049020ce7b/src/storage/src/hummock/sstable/forward_sstable_iterator.rs#L115

> The `skip_key_count` comes from ` __internal_nexmark_q7_1_hashjoinright_1007`. It was caused by `SELECT MAX(price) AS maxprice` ~~I think this is becuase we don't clean up the state after the window is...

Let's verify the metrics after #5522, #5531 and #5533 are merged.

> I think the cancellation is caused by no idle compactor, which is expected. I see. However, repeatedly triggering compaction and then cancelling it is kind of a waste of...

What can we gain from having graceful shutdown on CN? IMO, we anyway have to handle the case when a CN is hard killed (i.e. CN failure). If we can...

Can you also try this PR locally to verify the metric is correct in grafana dashboard before merging?

> I also observed that the object storage's `read-xx` metric dies with the compactor while some of the others do not. Should we start an new issue for it? @hzxa21...