exporters
exporters copied to clipboard
scrapinghub
Support list of filters
Right now, the library base export manager only support one filter before transforming, and one filter after transforming. I think we should support list of filters both before and after transforming. This would allow us to use different filtering approaches:
i.e. Right now, we can't filter all the items with "country_code"=="us" and remove duplicates at the same time.
I think this could be supported adding yet another filter class like MultipleFilter that allows you to specify combinations of filters.
@eliasdorneles What about always treating them as a "list of filters", just like we do with notifications. Of course, we would allow single filter configurations as we do now.
Filters would be applied FIFO, and an item only passes if all the filters pass. Thoughts @tsrdatatech?
My problem with a list of filters is that it's ambiguous (e.g. will the result be an AND or an OR?) -- essentially, the same problem we have currently with the KeyValue filters.
I prefer a new MultipleFilter class that allow you to choose combinations of these, allowing user to combine filters in a way like (FilterA AND FilterB) OR FilterC.
A good source of inspiration is MongoDB filtering API.
There is some discussion in https://github.com/scrapinghub/exporters/pull/312
My approach is mostly based on Mongo Query.
First of all I think that the filter key in the config should be a list, named filters to be clear.
After that I think that a filter composition could be simply as:
"filters": [
{"name": "exporters.filters.key_value_regex_filter.KeyValueRegexFilter",
"options": {
"keys": [
{"name": "country", "value": "United States"}
]
},
{"name": "exporters.filters.key_value_regex_filter.KeyValueRegexFilter",
"options": {
"keys": [
{"name": "city", "value": "New York"}
]
},
{"or": [
{"name": "exporters.filters.key_value_regex_filter.KeyValueRegexFilter",
"options": {
"keys": [
{"name": "country", "value": "Canada"}
]
},
{"name": "exporters.filters.key_value_regex_filter.KeyValueRegexFilter",
"options": {
"keys": [
{"name": "city", "value": "Montreal"}
]
},
]}
]
The config about could be translated to:
(country == 'United States' and city == 'New York') or (country == 'Canada' and city == 'Montreal'
You could also nest two or filters:
"filters": [
{"name": "exporters.filters.key_value_regex_filter.KeyValueRegexFilter",
"options": {
"keys": [
{"name": "country", "value": "United States"}
]
},
{"name": "exporters.filters.key_value_regex_filter.KeyValueRegexFilter",
"options": {
"keys": [
{"name": "city", "value": "New York"}
]
},
{"or": [
{"name": "exporters.filters.key_value_regex_filter.KeyValueRegexFilter",
"options": {
"keys": [
{"name": "country", "value": "Canada"}
]
},
{"or": [
{"name": "exporters.filters.key_value_regex_filter.KeyValueRegexFilter",
"options": {
"keys": [
{"name": "city", "value": "Montreal"}
]},
},
{"name": "exporters.filters.key_value_regex_filter.KeyValueRegexFilter",
"options": {
"keys": [
{"name": "status", "value": "Checked"}
]},
},
]}
]}
]
The config about could be translated to:
(country == 'United States' and city == 'New York') or (country == 'Canada' or (city == 'Montreal' and status == 'Checked'))
This is for sure tricky to implement, but I think that it's flexible and user-friendly enough. What do you guys think of this approach? cc: @eliasdorneles, @bbotella, @tsrdatatech
Well, I'll do a prof-of-concept of this idea, please, let me know if you guys have comments about that! :)
I have some work done in multiple_fitlers branch at my fork
Mostly of the code and test is done, @eliasdorneles I would appreciate if you can take a look and share with me your ideas. =)
I think tomorrow I'll update the docs and make the PR
@raphapassini please create a PR, it's hard to review & discuss in branches or lone commits.