scrapy-mysql-pipeline icon indicating copy to clipboard operation
scrapy-mysql-pipeline copied to clipboard

Make pipeline compatible with all item types

Open fkromer opened this issue 3 years ago • 2 comments

Introduce itemadapter into the pipeline to make the pipeline compatible with new item types pydanticobjects.

fkromer avatar Jun 07 '22 08:06 fkromer

Can you explain in more details your proposal?

IaroslavR avatar Jun 08 '22 11:06 IaroslavR

The logic to generate SQL statements from items is specific to items of type dict. There are some other item types supported in addition. Not documented in the official docs is support for items of type pydantic objects. This is the most interesting type cause they are the base class for a lot of other interesting libraries out there (SQLModel, etc.). As a result one could use models (which are based on SQLModel) as items and get rid of a lot of duplicated code.

itemadapters are used to provide a uniform API to work with items independent of their underlying implementation. Means one could add itemadapters into the package and convert any item to an adapter before accessing/mutating it.

fkromer avatar Jun 08 '22 11:06 fkromer