crawlee-python icon indicating copy to clipboard operation
crawlee-python copied to clipboard

Fix: Removed indentation from statistics logging

Open TymeeK opened this issue 1 year ago • 4 comments

Description

The purpose of the PR is to fix the indentation of statistics logging. It was originally 8 space indentation but now it is changed to be all on one line.

Issues

This fixes issue #306

Testing

I went through the unit test files and logged out what the output would be for the statistics loggin.

Example:

[crawlee.basic_crawler.basic_crawler] INFO  Final request statistics: {"requests_finished": 3, "requests_failed": 0, "retry_histogram": [3], "request_avg_failed_duration": null, "request_avg_finished_duration": 0.000984, "requests_finished_per_minute": 7348, "requests_failed_per_minute": 0, "request_total_duration": 0.002951, "requests_total": 3, "crawler_runtime": 0.024498}

Checklist

  • [ ] Changes are described in the CHANGELOG.md
  • [x] CI passed

TymeeK avatar Jul 17 '24 23:07 TymeeK

FYI we had a quick poll about this and in the end we agreed we want to print this as a formatted table instead of JSON, so we won't be merging this one.

Something like the following (but with snake_case instead of camelCase - this snippet was built based on the JS version)

┌──────────────────────────────────┬────────┐
│ requestsFinished                 │ 59     │
│ requestsFailed                   │ 0      │
│ retryHistogram                   │ [ 59 ] │
│ requestAvgFailedDurationMillis   │ null   │
│ requestAvgFinishedDurationMillis │ 1071   │
│ requestsFinishedPerMinute        │ 67     │
│ requestsFailedPerMinute          │ 0      │
│ requestTotalDurationMillis       │ 63215  │
│ requestsTotal                    │ 59     │
│ crawlerRuntimeMillis             │ 53197  │
└──────────────────────────────────┴────────┘

B4nan avatar Jul 18 '24 15:07 B4nan

We should be able to print tables using rich.Table - see https://rich.readthedocs.io/en/stable/tables.html

janbuchar avatar Jul 18 '24 15:07 janbuchar

Thanks for your feedback! I am brand new to open source. I did make some furthers changes by using a table to print it out as a formatted table. Should I open a new PR or update this one?

TymeeK avatar Jul 20 '24 00:07 TymeeK

Thanks for your feedback! I am brand new to open source. I did make some furthers changes by using a table to print it out as a formatted table. Should I open a new PR or update this one?

Awesome! Just commit the changes you made and push it to your fork and the PR will be updated automatically

janbuchar avatar Jul 20 '24 09:07 janbuchar