wg-prometheus icon indicating copy to clipboard operation
wg-prometheus copied to clipboard

Regarding accessing multiple workspaces

Open hgsat123 opened this issue 2 years ago • 1 comments

Hi, I have a created a workspace in a specified AWS region, provided necessary access to IAM role (annotated for a specified service account) to read/write access to this workspace. I'm able to write prometheus scrapped metrics (defined write_relabel_configs) to specified workspace. If i want to write same metrics to different workspace parallelly (mentioned url, sigv4 details of workspace region), prometheus ignores the second workspace though it exist in the same region (as first workspace). I see error - err="server returned HTTP status 404 Not Found: {"Message":"Workspace not found"}" in the prometheus logs.
I also see messages like Stopping remote storage...", Remote storage stopped." for the first workspace. So, prometheus shuts one of the workspace / storage to write WAL only to second one!. Question: Is there a way to write to multiple workspaces parallelly?

hgsat123 avatar Mar 10 '22 14:03 hgsat123

Hi, can you please provide some more detail so that we can understand your deployment and potential issue? Are you using the OpenTelemetry Collector? If so, can you provide your configuration? I would expect that you could define multiple prometheusremotewrite exporter instances and put them all on the same pipeline to achieve this result.

Aneurysm9 avatar Jul 12 '22 18:07 Aneurysm9