fix: #3146 Support close open/fail for Ready Tracker & surface errors swallowed by grp.Wait()

[David-Jaeyoon-Lee]

What this PR does / why we need it: This change fixes Run() method in Ready Tracker to have a fail-close option so that if we fail to fetch expectations we surface those errors & fail. By default, this won't be enabled & we will fail-open.

Which issue(s) this PR fixes (optional, using fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when the PR gets merged): Fixes #3146

Special notes for your reviewer: Refer to issue link for more information

[David-Jaeyoon-Lee]

Separating out constraint template v1beta1 -> v1 changes in separate PR.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 87.83784% with 36 lines in your changes missing coverage. Please review.

Project coverage is 47.83%. Comparing base (3350319) to head (bf49b3c). Report is 98 commits behind head on master.

Files	Patch %	Lines
pkg/syncutil/concurrent_slice.go	0.00%	19 Missing :warning:
pkg/readiness/ready_tracker.go	95.01%	11 Missing and 2 partials :warning:
pkg/syncutil/single_runner.go	71.42%	4 Missing :warning:

:exclamation: There is a different number of reports uploaded between BASE (3350319) and HEAD (bf49b3c). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (3350319)	HEAD (bf49b3c)
unittests	2	1

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3308      +/-   ##
==========================================
- Coverage   54.49%   47.83%   -6.67%     
==========================================
  Files         134      219      +85     
  Lines       12329    14884    +2555     
==========================================
+ Hits         6719     7120     +401     
- Misses       5116     6955    +1839     
- Partials      494      809     +315     

Flag	Coverage Δ
unittests	47.83% <87.83%> (-6.67%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[ritazh]

Thanks for the PR @David-Jaeyoon-Lee Few nits I did not see unit test added for the new flag.

[David-Jaeyoon-Lee]

@ritazh Here are the requested changes! let me know if there is anything else that needs to be changes :)

[maxsmythe]

May need to wrap fakeClient in a mutex to prevent the race condition, also there are lint errors

[David-Jaeyoon-Lee]

I still need to fix lint errors for the Error return value is not checked for the goroutine deleted object polling. Any ideas on how to fix this? @maxsmythe

[maxsmythe]

@David-Jaeyoon-Lee Unit tests are failing

[JaydipGabani]

Unit test passed, re-running to check for flakiness.

[JaydipGabani]

Running once more to check.