scrapy-mysql-pipeline
scrapy-mysql-pipeline copied to clipboard
Make pipeline compatible with all item types
Introduce itemadapter into the pipeline to make the pipeline compatible with new item types pydanticobjects.
Can you explain in more details your proposal?
The logic to generate SQL statements from items is specific to items of type dict. There are some other item types supported in addition. Not documented in the official docs is support for items of type pydantic objects. This is the most interesting type cause they are the base class for a lot of other interesting libraries out there (SQLModel, etc.). As a result one could use models (which are based on SQLModel) as items and get rid of a lot of duplicated code.
itemadapters are used to provide a uniform API to work with items independent of their underlying implementation. Means one could add itemadapters into the package and convert any item to an adapter before accessing/mutating it.