Ensure we recover from panics in all API originating paths

[simonswine]

We currently are not recovering panics in all cases, when we spawn up async processing in goroutines. That leaves esp. ingesters vulnerable to data loss after a panic.

I am proposing we check all relevant paths and consider replacing it with: https://github.com/kucherenkovova/safegroup