Add Topic1, Topic2, Topic3, LogsPerBlock fields to logpoller.Filter and log_poller_filters schema
Also in this PR:
- Converted all cases where a
logpoller.Filterobject is constructed over to using field names instead of positional arguments. This should make it much easier to add more fields in the future, and allow default values for uninteresting fields during tests to be filled in automatically. - Handle NULL HashArray and AddressArray reads from db by setting dest to nil instead of erroring out.
- Bug fix to SQL QUERY logging (handles NULL and empty array values for BYTEA[] now, instead of pancing)
- Minor improvements to SQL QUERY logging, so it's closer to an actual SQL query you can run via cut and paste
- Add test for writing and reading back in filters from db, with various combinations of topic filters set
I see that you haven't updated any README files. Would it make sense to do so?
Correct me if I'm wrong, but I don't see any usage of these filters during PollAndSave cycle. IIUC goal of this PR is to support filters with extended definition, but probably code responsible for using them during PollAndSave would be in a separate PR, right?
That's right. There are two followup PR's to make use of these extra fields. One of them changes the way in which logs are requested (instead of a single request with a big combined filter, it's a batch of requests sent all at once, normally 1 batch element per filter but with some optimizations for combining requests. This one is in progress, and there will be another PR after that for implementing the rate limiting & MaxLogsKept feature. Since both of these PR's required extending the logpoller.Filter object and migrating the db, I figured it made sense to combine that part of it as a common precursor to these changes.
Quality Gate passed
Issues
1 New issue
0 Fixed issues
0 Accepted issues
Measures
0 Security Hotspots
90.6% Coverage on New Code
0.0% Duplication on New Code