Kubeflow: Notebook App becomes unstable after deleting a lot of Notebooks

[jaystary]

Summary

When deleting a series of notebooks in rapid succession:

• it takes a long time for notebooks to be cleaned up
• in the inital phase the notebook app might become unreachable
• they get (re)initialized in the relevant NS (see video)
• I had instances in the past where this behavior led to a crash / restart of the entire cluster

Eventually (after 5-10mins) the cleanup finished and everything returns to normal.

I do think this is specific to Kubeflow as i observe this issue on various distributions but i havent been able so far to understand why this happens

Reproduction

Start 5-10 IDEs -> delete them, observe in Lens / KF UI

UI/UX Issue Screenshots

https://user-images.githubusercontent.com/34389140/154339950-98e23f30-cf58-4ba1-a254-2fc8a6b14ef6.mov

Additional Info about Your Environment

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We factor engagement into prioritization.