crawlee-python
crawlee-python copied to clipboard
Better format statistics logging
It seems we have 8 spaces indentation at the beginning:
[crawlee.statistics.statistics] INFO crawlee.beautifulsoup_crawler.beautifulsoup_crawler request statistics {
"requests_finished": 0,
"requests_failed": 0,
"retry_histogram": [
0
],
"request_avg_failed_duration": null,
"request_avg_finished_duration": null,
"requests_finished_per_minute": 0,
"requests_failed_per_minute": 0,
"request_total_duration": 0.0,
"requests_total": 0,
"crawler_runtime": 0.007741
}
[crawlee.beautifulsoup_crawler.beautifulsoup_crawler] INFO Final request statistics: {
"requests_finished": 32,
"requests_failed": 0,
"retry_histogram": [
32
],
"request_avg_failed_duration": null,
"request_avg_finished_duration": 0.349596,
"requests_finished_per_minute": 320,
"requests_failed_per_minute": 0,
"request_total_duration": 11.187069,
"requests_total": 32,
"crawler_runtime": 6.007066
}
How many spaces would we like for the indentation instead? 4?
@TymeeK Four would be great. I would like to get something like this:
[crawlee.beautifulsoup_crawler.beautifulsoup_crawler] INFO Final request statistics:
{
"requests_finished": 32,
"requests_failed": 0,
"retry_histogram": [
32
],
"request_avg_failed_duration": null,
"request_avg_finished_duration": 0.349596,
"requests_finished_per_minute": 320,
"requests_failed_per_minute": 0,
"request_total_duration": 11.187069,
"requests_total": 32,
"crawler_runtime": 6.007066
}
I would rather align this with how we do it in JS, so a single line instead. Here is an example of that:
2024-06-12T13:39:24.201Z INFO AdaptivePlaywrightCrawler: Final request statistics: {"requestsFinished":59,"requestsFailed":0,"retryHistogram":[59],"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":1071,"requestsFinishedPerMinute":67,"requestsFailedPerMinute":0,"requestTotalDurationMillis":63215,"requestsTotal":59,"crawlerRuntimeMillis":53197}
Now it looks like a forgotten log in the app instead of something done on purpose.