crawlee-python icon indicating copy to clipboard operation
crawlee-python copied to clipboard

Better format statistics logging

Open vdusek opened this issue 1 year ago • 3 comments

It seems we have 8 spaces indentation at the beginning:

[crawlee.statistics.statistics] INFO  crawlee.beautifulsoup_crawler.beautifulsoup_crawler request statistics {
        "requests_finished": 0,
        "requests_failed": 0,
        "retry_histogram": [
          0
        ],
        "request_avg_failed_duration": null,
        "request_avg_finished_duration": null,
        "requests_finished_per_minute": 0,
        "requests_failed_per_minute": 0,
        "request_total_duration": 0.0,
        "requests_total": 0,
        "crawler_runtime": 0.007741
      }
[crawlee.beautifulsoup_crawler.beautifulsoup_crawler] INFO  Final request statistics: {
        "requests_finished": 32,
        "requests_failed": 0,
        "retry_histogram": [
          32
        ],
        "request_avg_failed_duration": null,
        "request_avg_finished_duration": 0.349596,
        "requests_finished_per_minute": 320,
        "requests_failed_per_minute": 0,
        "request_total_duration": 11.187069,
        "requests_total": 32,
        "crawler_runtime": 6.007066
      }

vdusek avatar Jul 15 '24 14:07 vdusek

How many spaces would we like for the indentation instead? 4?

TymeeK avatar Jul 16 '24 01:07 TymeeK

@TymeeK Four would be great. I would like to get something like this:

[crawlee.beautifulsoup_crawler.beautifulsoup_crawler] INFO  Final request statistics:
{
    "requests_finished": 32,
    "requests_failed": 0,
    "retry_histogram": [
        32
    ],
    "request_avg_failed_duration": null,
    "request_avg_finished_duration": 0.349596,
    "requests_finished_per_minute": 320,
    "requests_failed_per_minute": 0,
    "request_total_duration": 11.187069,
    "requests_total": 32,
    "crawler_runtime": 6.007066
}

vdusek avatar Jul 16 '24 06:07 vdusek

I would rather align this with how we do it in JS, so a single line instead. Here is an example of that:

2024-06-12T13:39:24.201Z INFO  AdaptivePlaywrightCrawler: Final request statistics: {"requestsFinished":59,"requestsFailed":0,"retryHistogram":[59],"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":1071,"requestsFinishedPerMinute":67,"requestsFailedPerMinute":0,"requestTotalDurationMillis":63215,"requestsTotal":59,"crawlerRuntimeMillis":53197}

Now it looks like a forgotten log in the app instead of something done on purpose.

B4nan avatar Jul 16 '24 09:07 B4nan