Regression for 1.3.2: Pipeline Status did not get updated to set phase=Paused, drainedOnPause=true despite scaling down all vertices to 0

[juliev0]

Describe the bug

This is based on having run Numaplane using the rc 20 and rc 21 images that @whynowy built for me here.

What are these images? He built them by taking numaflow 1.3.2 and patching in my PR.

When set desiredPhase: Paused in the Pipeline after having updated the topology earlier, Numaflow actually scaled down all 3 vertices (in, cat, out) to 0 according to the log file and also seems to have updated currentPhase:Paused:

{"level":"info","ts":"2024-10-02T18:39:41.796000595Z","logger":"numaflow.controller-manager","caller":"pipeline/controller.go:194","msg":"Updated pipeline phase","namespace":"numaplane-system","pipeline":"test-pipeline-rollout-0","originalPhase":"Running","currentPhase":"Paused"}

Yet, there is no evidence in the watch that was done on the Pipeline that the phase in the log actually ever got set to Paused until the timeout occurred 30 seconds later.

Here are the numaplane log, numaflow log, and watch on all of the various CRs:

Archive.zip

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We often sort issues this way to know what to prioritize.

For quick help and support, join our slack channel.