clientconn: Wait for all goroutines on close

[twz123]

Three goroutines could outlive a call to ClientConn.close(). Add mechanics to cancel them and wait for them to complete when closing a client connection.

Fixes #8655.

RELEASE NOTES:

• Closing a client connection will cancel all pending goroutines and block until they complete.

[codecov[bot]]

Codecov Report

:x: Patch coverage is 85.93750% with 9 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 83.31%. Comparing base (38a9206) to head (40605a5).

Files with missing lines	Patch %	Lines
clientconn.go	76.31%	7 Missing and 2 partials :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8666      +/-   ##
==========================================
- Coverage   83.50%   83.31%   -0.19%     
==========================================
  Files         418      418              
  Lines       32485    32523      +38     
==========================================
- Hits        27125    27095      -30     
- Misses       3994     4036      +42     
- Partials     1366     1392      +26     

Files with missing lines	Coverage Δ
balancer_wrapper.go	85.34% <100.00%> (+1.30%)	:arrow_up:
internal/testutils/pipe_listener.go	85.71% <100.00%> (+1.93%)	:arrow_up:
clientconn.go	90.13% <76.31%> (-0.67%)	:arrow_down:

... and 25 files with indirect coverage changes

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[easwars]

@twz123 We had a discussion about this PR during our GitHub issues/PR scrub. Can I request you to split these into separate PRs. The change to the graceful_switch LB policy is much simpler compared to the changes in the clientconn for the other two goroutine leaks. We want to make progress on this as much as possible, but are also a little concerned about the complexity that it adds to the clientconn code (which is already reasonably complex).

Apologies for how long this review has dragged on, but hopefully once these are split into smaller ones, we should be able to move faster.

Thanks for your understanding.

[twz123]

@twz123 We had a discussion about this PR during our GitHub issues/PR scrub. Can I request you to split these into separate PRs. The change to the graceful_switch LB policy is much simpler compared to the changes in the clientconn for the other two goroutine leaks. We want to make progress on this as much as possible, but are also a little concerned about the complexity that it adds to the clientconn code (which is already reasonably complex).

Done: #8746

Apologies for how long this review has dragged on, but hopefully once these are split into smaller ones, we should be able to move faster.

Thanks for your understanding.

No worries. I understand that these things take time.

[easwars]

@twz123 Now that the change to the graceful switch balancer is merged, do you think what is remaining can be split up even further? Thanks.