Add tests for panics and context cancellation scenarios

[kohlisid]

          In the current setup, this is difficult to test via UT directly. I tried to add the test there but the assert.Panics (which internally uses recover) was not able to catch it as the panic is invoked from within another goroutine. Would need to research more on how can we cover something like this. In python i'm able to mock the panic and test which wasn't working out here.

I tried few manual scenarios to induce such panics which were working fine.

Originally posted by @kohlisid in https://github.com/numaproj/numaflow-go/pull/133#discussion_r1672695925