scrapy
Optimizing wrap_loader_context()
When working on a loader-heavy project I found that a lot of various inspect calls (and just a lot of various function calls) is done for every field of every loader. wrap_loader_context() calls get_func_args() for each processor which in turn does the most of the aforementioned things.
My tests show that a simple @lru_cache(1024) for get_func_args() is enough and that the cache in this case will contain one or several items for each processor used by the spider (so 1024 should be enough, though if it isn't, the cache will fail to give the performance improvements so this is debatable).
Another option would be changing get_value() and get_output_value() so that wrap_loader_context() isn't called this often, but get_value() takes the processors as input values.
Any reason to not merge a lru cache implementation? I came here after I noticed huge performance decrease for some spiders after updating to v1.1.0 from v1.0.6.
Some metrics for the most affected spider I could find:
| Version | items/second |
|---|---|
| v1.0.6 | 350 |
| v1.1.0 | 120 |
| v1.1.0 with lru | 550 |
