Error message for 'label value too long' misses important information

[bboreham]

The error seen by sender in Prometheus logs is like this:

level=warn ts=2019-07-22T09:25:16.899437956Z caller=queue_manager.go:530 component=remote queue=0:https://cloud.weave.works./api/prom/push msg="Error sending samples to remote storage" count=1000 err="server returned HTTP status 400 Bad Request: label value too long: \"{\\\"apiVersion\\\":\\\"extensions/v1beta1\\\",\\\"kind\\\":\\\"DaemonSet\\\",\\\"metadata\\\":{\\\"annotations\\\":{\\\"flux.weave.works/sync-checksum\\\":\\\"0b6f1382edb3f01d6a64cd775b46008b0f61003e\\\"},\\\"labels\\\":{\\\"flux.weave.works/sync-gc-mark\\\":\" metric \"kub"
level=warn ts=2019-07-22T09:25:32.00067315Z caller=queue_manager.go:530 component=remote queue=0:https://cloud.weave.works./api/prom/push msg="Error sending samples to remote storage" count=1000 err="server returned HTTP status 400 Bad Request: label value too long: \"{\\\"apiVersion\\\":\\\"extensions/v1beta1\\\",\\\"kind\\\":\\\"Deployment\\\",\\\"metadata\\\":{\\\"annotations\\\":{\\\"flux.weave.works/automated\\\":\\\"true\\\",\\\"flux.weave.works/automated_msg\\\":\\\"test\\\",\\\"flux.weave.works/automated_user\\\":\\\"Maarten\" metric \""

It looks like something is truncating the message just as it gets to the metric=... part. Maybe put that before the value? And it would also be useful to state the label name.

[tharun208]

I like to work on this @gouthamve

[bboreham]

As far as I can see the change in #4018 did not cover the "state the label name" part; reopening.