botasaurus icon indicating copy to clipboard operation
botasaurus copied to clipboard

Main task never ends in case of there is no result in google map.

Open iRajul opened this issue 5 months ago • 0 comments

how to reproduce

Search for string "botototootototototototototootototototoot"

it will give following error :

Traceback (most recent call last):
  File "C:\Python312\Lib\site-packages\botasaurus_server\task_executor.py", line 192, in run_task
    self.update_parent_task(task_id, result)
  File "C:\Python312\Lib\site-packages\botasaurus_server\retry_on_db_error.py", line 37, in wrapper
    raise e
  File "C:\Python312\Lib\site-packages\botasaurus_server\retry_on_db_error.py", line 26, in wrapper
    created_result = func(*args, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python312\Lib\site-packages\botasaurus_server\retry_on_db_error.py", line 77, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python312\Lib\site-packages\botasaurus_server\task_executor.py", line 205, in update_parent_task
    self.complete_parent_task_if_possible(parent_id, Server.get_remove_duplicates_by(scraper_name), result)
  File "C:\Python312\Lib\site-packages\botasaurus_server\task_executor.py", line 230, in complete_parent_task_if_possible
    fn()
  File "C:\Python312\Lib\site-packages\botasaurus_server\task_executor.py", line 227, in <lambda>
    fn = lambda: TaskHelper.read_clean_save_task(parent_id, remove_duplicates_by, status)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python312\Lib\site-packages\botasaurus_server\task_helper.py", line 190, in read_clean_save_task
    rs = normalize_dicts_by_fieldnames(TaskResults.get_all_task(parent_id))
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python312\Lib\site-packages\botasaurus_server\cleaners.py", line 35, in normalize_dicts_by_fieldnames
    for item in data:
TypeError: 'NoneType' object is not iterable
Error in run_task  'NoneType' object is not iterable

Potential Fix: Return an empty list from function normalize_dicts_by_fieldnames

def normalize_dicts_by_fieldnames(data):
    if data is None or len(data) == 0:
        return []

iRajul avatar Sep 13 '24 09:09 iRajul