Can't truncate manifest up to offset 21788466, offset of out range

[travisdowns]

Version & Environment

Redpanda version: v23.1.1

What went wrong?

The following error appears in the logs (extract from 3 different log files):

./redpanda-8/0.log.20230316-175605:2023-03-16T17:40:52.732532753Z stderr F ERROR 2023-03-16 17:40:52,732 [shard 13] cluster - ntp: {kafka/topic1/959} - archival_metadata_stm.cc:775 - Can't truncate manifest up to offset 21788466, offset out of range

./redpanda-10/0.log.20230316-175611:2023-03-16T17:40:52.674724875Z stderr F ERROR 2023-03-16 17:40:52,674 [shard 13] cluster - ntp: {kafka/topic1/959} - archival_metadata_stm.cc:775 - Can't truncate manifest up to offset 21788466, offset out of range

./redpanda-4/0.log.20230316-175609:2023-03-16T17:40:52.732560652Z stderr F ERROR 2023-03-16 17:40:52,732 [shard 13] cluster - ntp: {kafka/topic1/959} - archival_metadata_stm.cc:775 - Can't truncate manifest up to offset 21788466, offset out of range

Not that the error is from 3 different nodes, for the same ntp at around the same time (presumably, it's the 3 replicas for this partition).

Around this time the cluster was undergoing load tests and also topic deletion (of other topics).

What should have happened instead?

No errors.

How to reproduce the issue?

?

Additional information

I can provide the surrounding logfile if it is useful.

JIRA Link: CORE-1213