fantiadl icon indicating copy to clipboard operation
fantiadl copied to clipboard

Metadata check on paid and free plan scrape overwrites

Open bitbybyte opened this issue 4 years ago • 0 comments

For example, when a free plan user performs a scrape for a club already scraped with a paid plan, metadata will be overwritten. To fix this we can check the number of available post_contents to the scraping user and compare with the existing metadata. If more are available to the user, overwrite and continue downloading. If more are availble in the existing metadata, skip the post entirely.

We can also check which plan status is currently joined, or even simplify this entirely by specifying the type of metadata e.g. metadata_plan0.json.

bitbybyte avatar Mar 26 '20 20:03 bitbybyte