akka.net icon indicating copy to clipboard operation
akka.net copied to clipboard

Enable debug logging for ShardCoordinator/ShardRegion/Shard actors

Open LeonHartley opened this issue 1 year ago • 7 comments

Is your feature request related to a problem? Please describe. We are seeing potential issues with the shard coordinator when recovering millions of entities, where shard regions do not receive acknowledgement from the coordinator. We'd like to allow debug logs to be switched on for sharding system-level actors without having to enable debug logging for the whole system

Describe the solution you'd like A configuration switch to perhaps change the log level of all logs from the shard coordinator and shard region actors, so we can view more information, even when the rest of the system is set to Information or Warning level.

Describe alternatives you've considered We could switch debug logging on for the whole system but this would cause unnecessary stress on our logging infrastructure.

LeonHartley avatar Feb 14 '24 12:02 LeonHartley

There is a setting that will do this for Akka.Cluster.Sharding already, have you tried setting this to on ?

https://github.com/akkadotnet/akka.net/blob/f7ed2ac6c1baf88c8aacef8cc63a7c15de35d4fb/src/contrib/cluster/Akka.Cluster.Sharding/reference.conf#L239

Aaronontheweb avatar Feb 16 '24 17:02 Aaronontheweb

From the looks of it, that would still require us to enable debug logging for the whole system, this feature request is more around allowing debug logs without turning full system level log level to debug.

LeonHartley avatar Feb 17 '24 10:02 LeonHartley

I guess we could change that setting to accept a LOGLEVEL argument - that should do it

Aaronontheweb avatar Feb 17 '24 13:02 Aaronontheweb

Yeah that would work.

Cheers

LeonHartley avatar Feb 17 '24 18:02 LeonHartley

@LeonHartley so I implemented a fix for this in #7118 but decided not to merge it for 1.5.18 because it's kind of weird / sloppy.

What I think we're going to do instead is introduce a new "log filtering" feature on the backend that allows you to scrub out logs you don't care about at various log levels. I'll create a separate GH issue to spec it out, but I think that will be a much cleaner solution to your problems - it'll let you enable LogLevel.Debug and just say "get rid of everything that isn't from LogSources that start with /system/sharding/. Would that work for you?

Aaronontheweb avatar Mar 12 '24 17:03 Aaronontheweb

Yeah that sounds perfect, nice one @Aaronontheweb.

LeonHartley avatar Mar 12 '24 17:03 LeonHartley

Working on a preview of this feature here #7179

Aaronontheweb avatar Apr 29 '24 16:04 Aaronontheweb

@LeonHartley just merged this in - should go live with it in a day or so.

Aaronontheweb avatar May 22 '24 21:05 Aaronontheweb

Documentation for it will be live on the website here in about 20 minutes or so: https://getakka.net/articles/utilities/logging.html - at the bottom of the page

Aaronontheweb avatar May 22 '24 21:05 Aaronontheweb