Scrape fails when model has a large amount of content

[jimcrapo]

Example - this particular user has 474 photos, 115 videos, 110 archived posts

Scrape Processing Name: INFO [alembic.runtime.migration] Context impl SQLiteImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. INFO [alembic.runtime.migration] Running upgrade 6b1b10eb67de -> a918b6b05d2f, content INFO [alembic.runtime.migration] Running upgrade a918b6b05d2f -> 990fc1108317, content INFO [alembic.runtime.migration] Context impl SQLiteImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. INFO [alembic.runtime.migration] Running upgrade 6b1b10eb67de -> a918b6b05d2f, content INFO [alembic.runtime.migration] Running upgrade a918b6b05d2f -> 990fc1108317, content Type: Profile Type: Stories No Stories Found. Type: Posts Scrape Attempt: 1/100 Type: Archived Posts Scrape Attempt: 1/100 Traceback (most recent call last): File "F:\rip\start_ofd.py", line 60, in loop.run_until_complete(main()) File "C:\Python\Python39\lib\asyncio\base_events.py", line 642, in run_until_complete return future.result() File "F:\rip\start_ofd.py", line 44, in main api = await main_datascraper.start_datascraper( File "F:\rip\datascraper\main_datascraper.py", line 93, in start_datascraper await main_helper.process_names( File "F:\rip\helpers\main_helper.py", line 939, in process_names result = await module.start_datascraper(authed, username, site_name) File "F:\rip\modules\onlyfans.py", line 160, in start_datascraper await prepare_scraper(authed, site_name, item) File "F:\rip\modules\onlyfans.py", line 793, in prepare_scraper master_set += await subscription.get_archived_posts() File "F:\rip\apis\onlyfans\classes\create_user.py", line 427, in get_archived_posts final_results = [create_post(x, self) for x in results if x] File "F:\rip\apis\onlyfans\classes\create_user.py", line 427, in final_results = [create_post(x, self) for x in results if x] File "F:\rip\apis\onlyfans\classes\create_post.py", line 13, in init self.author = create_user.create_user(option["author"]) KeyError: 'author'

[oz-richardson-reclique]

I'm having the same issue with large profiles.

[OFfriend]

please use https://github.com/DIGITALCRIMINALS/OnlyFans