Is possible to use bloom filter on scrapy-redis?

[zhangslob]

I offen come up a situation using scrapy-redis, with too many dupefilters in redis, may be 1 trillions or more.My redis has only 16GB...

Others has made ScrapyRedisBloomFilter which is only for scrapy but not for scrapy-redis. So how can I use BloomFilter in scrapy-redis?

Looking forward to your reply~

[LIpengcccc]

可以的，大佬，我好像读了你的知乎 链接：https://pan.baidu.com/s/1hryRHoe8GM1kpy4aBrq7nw 提取码：j5iu ，这个好像讲了

[zhangslob]

@LIpengcccc Thanks!

[bqlwx111]

The bloom filter requires your own implementation. It only provides redis connections and there is no way to use rebloom directly, 布隆过滤器需要你自己实现，它只是提供redis连接，也没有办法直接使用rebloom

[leffss]

https://github.com/leffss/ScrapyRedisBloomFilterBlockCluster