chainlink icon indicating copy to clipboard operation
chainlink copied to clipboard

Add Topic1, Topic2, Topic3, LogsPerBlock fields to logpoller.Filter and log_poller_filters schema

Open reductionista opened this issue 1 year ago • 3 comments

Also in this PR:

  • Converted all cases where a logpoller.Filter object is constructed over to using field names instead of positional arguments. This should make it much easier to add more fields in the future, and allow default values for uninteresting fields during tests to be filled in automatically.
  • Handle NULL HashArray and AddressArray reads from db by setting dest to nil instead of erroring out.
  • Bug fix to SQL QUERY logging (handles NULL and empty array values for BYTEA[] now, instead of pancing)
  • Minor improvements to SQL QUERY logging, so it's closer to an actual SQL query you can run via cut and paste
  • Add test for writing and reading back in filters from db, with various combinations of topic filters set

reductionista avatar Feb 07 '24 01:02 reductionista

I see that you haven't updated any README files. Would it make sense to do so?

github-actions[bot] avatar Feb 07 '24 01:02 github-actions[bot]

Correct me if I'm wrong, but I don't see any usage of these filters during PollAndSave cycle. IIUC goal of this PR is to support filters with extended definition, but probably code responsible for using them during PollAndSave would be in a separate PR, right?

That's right. There are two followup PR's to make use of these extra fields. One of them changes the way in which logs are requested (instead of a single request with a big combined filter, it's a batch of requests sent all at once, normally 1 batch element per filter but with some optimizations for combining requests. This one is in progress, and there will be another PR after that for implementing the rate limiting & MaxLogsKept feature. Since both of these PR's required extending the logpoller.Filter object and migrating the db, I figured it made sense to combine that part of it as a common precursor to these changes.

reductionista avatar Feb 21 '24 22:02 reductionista