backstage Make catalog logging a little quieter where entities are erroring

Hey, I just made a Pull Request!

When an error is raised by a url reader or any other processor in the catalog, those errors are logged to the application logs with a warning level as well as written to the refresh_state table for retrieval later.

Given that these errors are already visible at the application layer via the refresh state table, I would argue that the log line is superfluous. The log line can be useful in some cases so I have left it in at debug level.

In the end this change makes the catalog logging a little quieter where entities are erroring.

I dont feel terribly strongly about this, and am happy to close this if there is any contention.

:heavy_check_mark: Checklist

[x] A changeset describing the change and affected packages. (more info)
[ ] Added or updated documentation
[ ] Tests for new functionality and regression tests for bug fixes
[ ] Screenshots attached (for UI changes)
[x] All your commits have a Signed-off-by line in the message. (more info)

Feb 16 '24 13:02 punkle

Changed Packages

Package Name	Package Path	Changeset Bump	Current Version
@backstage/plugin-catalog-backend	plugins/catalog-backend	patch	`v1.21.0`

Feb 16 '24 13:02 backstage-goalie[bot]

Uffizzi Cluster pr-23022 was deleted.

Feb 16 '24 13:02 github-actions[bot]

For a long time while I was at Keyloop we had no access to the database, I think this is true for other adopters as well. Also, for early Adopters these are errors they will run into first as they learn. I feel like moving them to debug makes them harder to surface especially as turning on the debug logs creates very verbose output making it harder to catch things like this.

That said, happy to go with what every the Maintainer decide on this 👍

Feb 16 '24 13:02 awanlin

@awanlin The errors are shown in the application in a few places:

InspectEntityDialog, you can get there from the three dots in the top right of the catalog item.
The plugin gives you an overview of the processing errors also: @backstage/plugin-catalog-unprocessed-entities

Feb 16 '24 14:02 punkle

My concern here is this makes integrating with log based tooling a lot more difficult. We use logs to generate both metrics and alerting, and while this would be possible by enabling debug logging, that would also generate a lot of noise and cost when it comes to storing those logs.

Feb 22 '24 16:02 mikeyhc

Can you elaborate a little? Which metrics and alerts are you triggering from this log line?

Feb 23 '24 07:02 punkle

Sure! We use this to generate metrics and alert around total number of erroring entities. Seeing a steep uptick in this is often indicative of a new automation generating invalid entities, and has helped us proactively reached out to catalog users. Also as a large number of erroring entities can negatively impact processing times it has allowed us to head off incidents related to that.

Feb 23 '24 15:02 mikeyhc

@mikeyhc I suppose what we could do to satisfy both concerns is to emit an event on the event broker and then make a configurable event subscriber that writes a log line or a produces a metric or something..

Then if you dont want the log line, dont configure the subscriber. If you do want the log line, configure the subscriber. Something like that.

Would that work?

Feb 26 '24 09:02 punkle

@punkle sounds good to me

Feb 26 '24 14:02 mikeyhc

@mikeyhc Ive implemented that suggestion in the latest version of this pull request. Let me know if you have any feedback on the specifics.

Feb 26 '24 14:02 punkle