redash icon indicating copy to clipboard operation
redash copied to clipboard

Consolidate query execution logs

Open zachliu opened this issue 1 year ago • 4 comments
trafficstars

What type of PR is this?

  • [x] Refactor
  • [ ] Feature
  • [ ] Bug Fix
  • [ ] New Query Runner (Data Source)
  • [ ] New Alert Destination
  • [ ] Other

Description

Redundant log entries are removed. All information (in-memory result size, query runtime, etc.) is consolidated under the _log_progress() method.

Another benefit of consolidated log entries is that when using tools such as DataDog, custom metrics can be easily generated from parsed logs:

dd

Because username and query_id are now tied with data_length and run_time by putting them in the same log entry. Much easier to track rogue users who like to do SELECT * FROM big_table or any other inefficient queries.

How is this tested?

  • [ ] Unit tests (pytest, jest)
  • [ ] E2E Tests (Cypress)
  • [x] Manually
  • [ ] N/A

Related Tickets & Documents

Mobile & Desktop Screenshots/Recordings (if there are UI changes)

zachliu avatar Aug 07 '24 18:08 zachliu

The concept of this sounds good. I'll have to look at the implementation later on after some other tasks though, if no-one else gets to it first.

@lucasfcnunes Is this your kind of thing to review the technical details of? :smile:

justinclift avatar Sep 17 '24 03:09 justinclift

😆 I'll take a look

lucasfcnunes avatar Sep 17 '24 18:09 lucasfcnunes

@lucasfcnunes @justinclift gently nudge :wink:

zachliu avatar Apr 29 '25 16:04 zachliu

@lucasfcnunes thanks! :pray:

zachliu avatar Apr 30 '25 14:04 zachliu