callbackserializer: add a method to execute a callback in a blocking fashion

[easwars]

Currently, we have a couple of ways to schedule callbacks for execution with the serializer:

• TrySchedule: which is a best effort option to schedule a callback
• ScheduleOr: which provides a way for the caller to supply a function that is executed if the callback is not successfully scheduled on the serializer.

See: https://github.com/grpc/grpc-go/blob/master/internal/grpcsync/callback_serializer.go

We have a whole bunch of places which have the following pattern (where the caller wants to wait for the scheduled callback to run, before proceeding):

	done := make(chan struct{})
	serializer.ScheduleOr(func(context.Context) {
		// Do stuff
		close(done)
	}, func() {
		// Set an error somewhere or throw a log message
		close(done)
    })
	<-done

We could introduce a new method to the serializer which supports this pattern of blocking until the provided callback is run. This review thread has some ideas to get started on this: https://github.com/grpc/grpc-go/pull/8499#discussion_r2257962052

[turfaa]

Quoting from the mentioned thread:

Maybe even something nice with generics:

// Schedules f and waits for it to be executed.  Returns what T returned or nil and
// ErrSerializerClosedBeforeItCouldRunFOrWhatever.
func (cs *CallbackSerializer) ScheduleAndWait[T any](f func(ctx context.Context) T) (T, error)

I don't think this is possible? The type parameter would need to be declared as the struct's type parameter. This would hugely limit CallbackSerializer.ScheduleAndWait's capabilities.

If returning a value is preferable, it is possible to make

func (cs *CallbackSerializer) ScheduleAndWait(f func(ctx context.Context) any) (any, error)

However it would probably require the callers to do type casting, or even possibly type assertion for safety.

[colin-si]

I think this method is still necessary to provide, which can eliminate a lot of boilerplate code and reduce potential errors. In many usage scenarios, it is necessary to block and wait for the callback to complete