Redpanda fails to stop after topic recovery

[LenaAn]

Version & Environment

Redpanda version: (use rpk version): latest (rev 8de704b) I'm using debug build, dev branch, commit 8de704b87b6dfa23d9c0dbd28fbaadb02c254809

• Operating System (e.g. from /etc/os-release):

PRETTY_NAME="Ubuntu 21.10"
NAME="Ubuntu"
VERSION_ID="21.10"
VERSION="21.10 (Impish Indri)"
VERSION_CODENAME=impish
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=impish

What went wrong?

I started a 3-nodes cluster, recovered a topic using rpk topic create <topic name> -r 1 -p 1 -c cleanup.policy=delete -c retention.bytes=999999999999 -c redpanda.remote.recovery=true, stopped nodes 2 and 1 by Ctrl+c, tried to stop node 0 Ctrl+c, but it stalled.

^CINFO  2022-08-03 13:40:32,235 [shard 0] redpanda::main - application.cc:250 - Stopping...
INFO  2022-08-03 13:40:32,235 [shard 0] rpc - server.cc:252 - kafka rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,236 [shard 0] rpc - server.cc:261 - kafka rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,236 [shard 12] rpc - server.cc:252 - kafka rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,236 [shard 14] rpc - server.cc:252 - kafka rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,236 [shard 13] rpc - server.cc:252 - kafka rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,237 [shard 12] rpc - server.cc:261 - kafka rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,237 [shard 14] rpc - server.cc:261 - kafka rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,237 [shard 2] rpc - server.cc:252 - kafka rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,237 [shard 13] rpc - server.cc:261 - kafka rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,237 [shard 2] rpc - server.cc:261 - kafka rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,237 [shard 7] rpc - server.cc:252 - kafka rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,237 [shard 10] rpc - server.cc:252 - kafka rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,237 [shard 7] rpc - server.cc:261 - kafka rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,237 [shard 9] rpc - server.cc:252 - kafka rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,237 [shard 10] rpc - server.cc:261 - kafka rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,237 [shard 3] rpc - server.cc:252 - kafka rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,237 [shard 9] rpc - server.cc:261 - kafka rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,237 [shard 11] rpc - server.cc:252 - kafka rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,237 [shard 3] rpc - server.cc:261 - kafka rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,237 [shard 11] rpc - server.cc:261 - kafka rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,237 [shard 5] rpc - server.cc:252 - kafka rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,237 [shard 15] rpc - server.cc:252 - kafka rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,238 [shard 4] rpc - server.cc:252 - kafka rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,238 [shard 15] rpc - server.cc:261 - kafka rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,238 [shard 5] rpc - server.cc:261 - kafka rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,238 [shard 6] rpc - server.cc:252 - kafka rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,238 [shard 4] rpc - server.cc:261 - kafka rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,238 [shard 6] rpc - server.cc:261 - kafka rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,238 [shard 1] rpc - server.cc:252 - kafka rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,238 [shard 8] rpc - server.cc:252 - kafka rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,238 [shard 1] rpc - server.cc:261 - kafka rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,238 [shard 8] rpc - server.cc:261 - kafka rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,238 [shard 0] rpc - server.cc:252 - vectorized internal rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,238 [shard 0] rpc - server.cc:261 - vectorized internal rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,239 [shard 3] rpc - server.cc:252 - vectorized internal rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,240 [shard 3] rpc - server.cc:261 - vectorized internal rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,240 [shard 10] rpc - server.cc:252 - vectorized internal rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,240 [shard 12] rpc - server.cc:252 - vectorized internal rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,240 [shard 10] rpc - server.cc:261 - vectorized internal rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,240 [shard 12] rpc - server.cc:261 - vectorized internal rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,240 [shard 1] rpc - server.cc:252 - vectorized internal rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,240 [shard 6] rpc - server.cc:252 - vectorized internal rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,240 [shard 8] rpc - server.cc:252 - vectorized internal rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,240 [shard 1] rpc - server.cc:261 - vectorized internal rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,240 [shard 8] rpc - server.cc:261 - vectorized internal rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,240 [shard 6] rpc - server.cc:261 - vectorized internal rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,240 [shard 14] rpc - server.cc:252 - vectorized internal rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,240 [shard 13] rpc - server.cc:252 - vectorized internal rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,240 [shard 14] rpc - server.cc:261 - vectorized internal rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,240 [shard 13] rpc - server.cc:261 - vectorized internal rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,240 [shard 7] rpc - server.cc:252 - vectorized internal rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,240 [shard 2] rpc - server.cc:252 - vectorized internal rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,240 [shard 2] rpc - server.cc:261 - vectorized internal rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,240 [shard 7] rpc - server.cc:261 - vectorized internal rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,240 [shard 15] rpc - server.cc:252 - vectorized internal rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,240 [shard 5] rpc - server.cc:252 - vectorized internal rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,241 [shard 11] rpc - server.cc:252 - vectorized internal rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,241 [shard 5] rpc - server.cc:261 - vectorized internal rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,241 [shard 15] rpc - server.cc:261 - vectorized internal rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,241 [shard 11] rpc - server.cc:261 - vectorized internal rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,241 [shard 9] rpc - server.cc:252 - vectorized internal rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,241 [shard 9] rpc - server.cc:261 - vectorized internal rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,241 [shard 4] rpc - server.cc:252 - vectorized internal rpc protocol - Stopping 1 listeners
INFO  2022-08-03 13:40:32,241 [shard 4] rpc - server.cc:261 - vectorized internal rpc protocol - Shutting down 0 connections
INFO  2022-08-03 13:40:32,377 [shard 0] archival - [fiber1] - service.cc:171 - Scheduler service stop
INFO  2022-08-03 13:40:32,378 [shard 12] archival - [fiber1] - service.cc:171 - Scheduler service stop
INFO  2022-08-03 13:40:32,378 [shard 3] archival - [fiber1] - service.cc:171 - Scheduler service stop
INFO  2022-08-03 13:40:32,378 [shard 15] archival - [fiber1] - service.cc:171 - Scheduler service stop
INFO  2022-08-03 13:40:32,378 [shard 11] archival - [fiber1] - service.cc:171 - Scheduler service stop
INFO  2022-08-03 13:40:32,378 [shard 13] archival - [fiber1] - service.cc:171 - Scheduler service stop
INFO  2022-08-03 13:40:32,379 [shard 9] archival - [fiber1] - service.cc:171 - Scheduler service stop
INFO  2022-08-03 13:40:32,379 [shard 14] archival - [fiber1] - service.cc:171 - Scheduler service stop
INFO  2022-08-03 13:40:32,379 [shard 2] archival - [fiber1] - service.cc:171 - Scheduler service stop
INFO  2022-08-03 13:40:32,379 [shard 7] archival - [fiber1] - service.cc:171 - Scheduler service stop
INFO  2022-08-03 13:40:32,379 [shard 6] archival - [fiber1] - service.cc:171 - Scheduler service stop
INFO  2022-08-03 13:40:32,379 [shard 8] archival - [fiber1] - service.cc:171 - Scheduler service stop
INFO  2022-08-03 13:40:32,379 [shard 5] archival - [fiber1] - service.cc:171 - Scheduler service stop
INFO  2022-08-03 13:40:32,380 [shard 4] archival - [fiber1] - service.cc:171 - Scheduler service stop
INFO  2022-08-03 13:40:32,380 [shard 1] archival - [fiber1] - service.cc:171 - Scheduler service stop
INFO  2022-08-03 13:40:32,387 [shard 10] archival - [fiber1] - service.cc:171 - Scheduler service stop
INFO  2022-08-03 13:40:32,447 [shard 0] raft - [group_id:0, {redpanda/controller/0}] consensus.cc:238 - Stopping
^C^CINFO  2022-08-03 13:40:35,089 [shard 0] cluster - error refreshing cluster health state - Currently there is no leader controller elected in the cluster
WARN  2022-08-03 13:40:35,089 [shard 0] cluster - health_monitor_backend.cc:445 - Failed to refresh cluster health.
INFO  2022-08-03 13:40:40,090 [shard 0] cluster - error refreshing cluster health state - Currently there is no leader controller elected in the cluster
WARN  2022-08-03 13:40:40,090 [shard 0] cluster - health_monitor_backend.cc:445 - Failed to refresh cluster health.
^CINFO  2022-08-03 13:40:45,091 [shard 0] cluster - error refreshing cluster health state - Currently there is no leader controller elected in the cluster
WARN  2022-08-03 13:40:45,091 [shard 0] cluster - health_monitor_backend.cc:445 - Failed to refresh cluster health.

Last two lines are repeated every 5 seconds.

What should have happened instead?

Redpanda should stop by Ctrl+C

How to reproduce the issue?

Additional information

Please attach any relevant logs, backtraces, or metric charts.

[LenaAn]

looks like partition_downloader::download_segment_file doesn't stop. offset_translator::copy_stream calls storage::copy_helper::run() which doesn't stop until it copies the whole segment.

@Lazin do you have ideas how to fix it?