Handle scaling events for special failed/other cases

[kohlisid]

Summary

In certain scenarios like exceeding resource quota the pipeline vertex/monovertex goes into a failed state failing to reconcile. If this persists, there can be conditions where we would want to scale down, but the autoscaler will always skip this as the vertex remains in failed state. We might want to consider having some special conditions where we can take some actions on a failed phase during scaling.

Use Cases

When would you use this?

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.