Added duration in seconds as logger attribute

[or-shachar]

I'm trying to monitor reconciliation durations using the structured log objects, and I found it hard to extract it out of the plain message. I suggest adding another field to the log object with durationSeconds that can be easily turned into a metric.

I'm not quite sure how to test this new code though, seems like the logging is uncovered by the tests anyhow. Any advice would be appreciated!

[or-shachar]

@mvoitko Sorry for the late response. Marked this as ready to review and rebased the branch on top of main.

[stefanprodan]

We have a Prometheus histogram for this, please see the docs here https://fluxcd.io/docs/guides/monitoring/#metrics

If we decide to add the durationSeconds to structured logs, then this must be done across all Flux controllers.

[or-shachar]

@stefanprodan - Actually when I opened it we didn't have prometheus scraper within the cluster but only logs scraper. We used a different solution for metric reporting.

Now we do use Prometheus so it's not that important for us and indeed we use that historgram.

Do you think it's still valuable to do it across all Flux controllers? The pro is that it allows collecting data even without Prometheus. I don't know if our use case was unique or not.