frontera icon indicating copy to clipboard operation
frontera copied to clipboard

Metadata is not saved to db in single process mode

Open Prometheus3375 opened this issue 5 years ago • 2 comments

Metadata is saved in distributed mode if there is a db worker with no flag --no-incoming. When I switched to single process mode, metadata is not saved. I did not find any setting that enables it. Is it indented? According to documentation metadata should be saved.

image

Here you can find my project. I am using discovery strategy. To add seeds, run python -m frontera.utils.add_seeds --config config.single --seeds-file seeds.txt. To start crawl, run scrapy crawl spider.

Prometheus3375 avatar Jun 12 '20 17:06 Prometheus3375

Hi, is sqlite file created, is anything written there?

sibiryakov avatar Jun 18 '20 08:06 sibiryakov

@sibiryakov, yes, queue, states and domain metadata tables are filling with data. Only metadata is always empty in single process mode.

Prometheus3375 avatar Jun 19 '20 09:06 Prometheus3375