dCache don't clean up the old requests from old pools

[qiulan2021]

trafficstars

Hello dCache Team,

From our dcache instance, we observed clients retry the stuck requests and completed, while the old requests were not clean up by dcache from the old pools. The kind of requests are indeed not in PoolManager but stay in the pools, which causes zombie requests.

The zombie requests should be gone from old pools, otherwise the daemon will still process it until it's done.

[dcadmin02] (dc226_38@dc226thirtyeightDomain) admin > rh ls
2955360d-d49a-43cc-b282-364eec1d6c09 ACTIVE Thu Sep 22 11:38:36 EDT 2022 Thu Sep 22 11:38:36 EDT 2022 0000AF66AD6951C64BBA821A64EA7DDAE52E MCTAPE:MC
8761c9e7-2357-4854-be15-3bf2de2bd54d ACTIVE Thu Sep 22 11:38:36 EDT 2022 Thu Sep 22 11:38:36 EDT 2022 0000345274BE1B274DDCA8258B7F0A461A38 MCTAPE:MC
0be1beaa-f296-4238-b49c-1f5ba2e3d92d ACTIVE Thu Sep 22 11:38:36 EDT 2022 Thu Sep 22 11:38:36 EDT 2022 00004028EFB6734D41CA84375BD20016A0F7 MCTAPE:MC
0c76c383-4dd3-4f03-9038-904a9677e64b ACTIVE Thu Sep 22 11:38:36 EDT 2022 Thu Sep 22 11:38:36 EDT 2022 0000A65F5AC722E54437839266294D5C1AEE MCTAPE:MC
34206efc-8204-438f-813f-7240d883dd61 ACTIVE Thu Sep 22 11:38:36 EDT 2022 Thu Sep 22 11:38:36 EDT 2022 0000BAD87541C2F840C2AD0922AE17B48C03 MCTAPE:MC
e37496c3-fa8c-449c-80f5-7d86287ef7c4 ACTIVE Thu Sep 22 11:38:36 EDT 2022 Thu Sep 22 11:38:36 EDT 2022 00009849B9F046854E3F9A393EB9C8811D3C MCTAPE:MC
df7045f5-bf9f-430c-ba8d-8b4029716ca3 ACTIVE Thu Sep 22 11:38:36 EDT 2022 Thu Sep 22 11:38:36 EDT 2022 000050DE756457CC4A75986B79B9C3826CF6 MCTAPE:MC
8340135c-2dad-49aa-8b2a-8e91ed9939fb ACTIVE Thu Sep 22 11:38:36 EDT 2022 Thu Sep 22 11:38:36 EDT 2022 00003EC155113F6F416E9477E709628B3627 MCTAPE:MC
068955a8-0679-436d-9303-cfb9170d470a ACTIVE Thu Sep 22 11:38:36 EDT 2022 Thu Sep 22 11:38:36 EDT 2022 00008DBEB5F097F44810A1845EB98A1BDC63 MCTAPE:MC
43be2a96-3477-41af-80bf-f986f8742ead ACTIVE Thu Sep 22 11:38:36 EDT 2022 Thu Sep 22 11:38:36 EDT 2022 00008C9D5CC0B7544522A7C37FE7FE50CDEB MCTAPE:MC
b6e22ee4-415a-4039-b3f4-79a2edd38a18 ACTIVE Thu Sep 22 11:38:36 EDT 2022 Thu Sep 22 11:38:36 EDT 2022 0000B010B45AFDDE4D8398DCC035590D0FB0 MCTAPE:MC
7b4acdfe-2a5f-4dc8-8073-b3838fa43cf1 ACTIVE Thu Sep 22 11:38:36 EDT 2022 Thu Sep 22 11:38:36 EDT 2022 000054A0C90888554C8EBBAA3FE1E2E342C2 MCTAPE:MC
0cc99d0a-224e-41bc-bdc9-dd45208e3c12 ACTIVE Thu Sep 22 11:38:36 EDT 2022 Thu Sep 22 11:38:36 EDT 2022 0000E2864E1302E646D7AF0B1245789FB7D6 MCTAPE:MC
e20fb231-b812-4cd0-b9f6-935c363171ee ACTIVE Thu Sep 22 11:38:36 EDT 2022 Thu Sep 22 11:38:36 EDT 2022 0000D4167B2D008D481D89AB5DE006C11D34 MCTAPE:MC
bb6cfe4b-3f64-491c-b8b3-bedfc6745c72 ACTIVE Thu Sep 22 11:38:36 EDT 2022 Thu Sep 22 11:38:36 EDT 2022 0000A4B0DAF1A3BD4D8DBEDAC74E4368D4D1 MCTAPE:MC
8460d0e6-8cf3-4877-bf79-c6b1fdedb204 ACTIVE Thu Sep 22 11:38:36 EDT 2022 Thu Sep 22 11:38:36 EDT 2022 00006D5FB947A6A14333818F82919A9BC27D MCTAPE:MC
cc60519b-2e97-460d-9a8f-6e3b22664633 ACTIVE Thu Sep 22 11:38:36 EDT 2022 Thu Sep 22 11:38:36 EDT 2022 0000F03E9F0FF50A4733BF0CB348298979B4 MCTAPE:MC

[dcadmin02] (PoolManager@dCacheDomain) admin > rc ls
[email protected]/0-*/* m=1 r=0 [dc213_38] [Waiting for stage: dc213_38 09.23 14:44:39] {0,}
[email protected]/0-*/* m=1 r=0 [dc215_38] [Waiting for stage: dc215_38 09.23 14:46:16] {0,}