High number of Image download tokens operations stuck in running state

[FrancescoDeSimone]

Required information

• Distribution: Ubuntu
• Distribution version: 22.04.4 LTS"
• The output of "snap list --all lxd core20 core22 core24 snapd":

Name    Version         Rev    Tracking       Publisher   Notes
core20  20240227        2264   latest/stable  canonical✓  base,disabled
core20  20240416        2318   latest/stable  canonical✓  base
core22  20240111        1122   latest/stable  canonical✓  base,disabled
core22  20240408        1380   latest/stable  canonical✓  base
lxd     5.21.1-10f4115  28322  latest/stable  canonical✓  in-cohort,held
snapd   2.61.2          21184  latest/stable  canonical✓  snapd,held
snapd   2.62            21465  latest/stable  canonical✓  snapd,disabled,held

• The output of "lxc info" or if that fails:
  • Kernel version:
  • LXC version:
  • LXD version:
  • Storage backend in use:

https://pastebin.ubuntu.com/p/WrGFywnTRr/plain/

Issue description

After deploying microcloud an high number of Image download token operation is create. lxc operation list I have 378 line with Image download token

output of lxc operation list: https://pastebin.canonical.com/p/8CMxXG4nXh/ Update 07/06/2024: I have not registered new operation on the first microcloud node since a week: https://pastebin.canonical.com/p/NGFnWn3Fz6/plain/

On the other 2 microcloud node now I have new operation regarding the first node microcloud node2: https://pastebin.canonical.com/p/DWtzVP7pYM/plain/ microcloud node3: https://pastebin.canonical.com/p/gbFW4vPtmk/plain/

Steps to reproduce

Information to attach

• [ ] Any relevant kernel output (dmesg)
• [ ] Container log (lxc info NAME --show-log)
• [ ] Container configuration (lxc config show NAME --expanded)
• [ ] Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
• [ ] Output of the client with --debug
• [ ] Output of the daemon with --debug (alternatively output of lxc monitor while reproducing the issue)