python-mapswipe-workers icon indicating copy to clipboard operation
python-mapswipe-workers copied to clipboard

wrong task in results leads to ForeignKeyViolation

Open Hagellach37 opened this issue 2 years ago • 0 comments

When there is a result such as the one below in Firebase, this will lead to an error when transfering the results to postgres. This is linked to this issue on the client side https://github.com/mapswipe/mapswipe/issues/619

The task t34299 belongs to another group and triggers the Foreign Key violation later on.

A very simple approach could look like this (it's more or less what I do manually):

  • first try to just transfer the data without any check
  • if the transfer fails --> get the information from the error log which group and task is affected
  • remove the wrong task for the affected group
  • try to run the transfer again
  • repeat if it still fails

There are probably nicer ways to do this, e.g.

  • import results to table results_temp (this table does not have any keys defined)
  • check the results_temp table for potential ForeignKeyViolation
  • remove the all faulty results from results_temp
  • insert results_temp into results
  "endTime" : "2022-03-25T22:06:24.578Z",
  "results" : {
    "t34299" : 0,
    "t35150" : 0,
    "t35151" : 0,
    "t35152" : 0,
    "t35153" : 0,
    "t35154" : 0,
    "t35155" : 0,
    "t35156" : 0,
    "t35157" : 0,
    "t35158" : 0,
    "t35159" : 0,
    "t35160" : 0,
    "t35161" : 0,
    "t35162" : 0,
    "t35163" : 0,
    "t35164" : 0,
    "t35165" : 0,
    "t35166" : 0,
    "t35167" : 0,
    "t35168" : 0,
    "t35169" : 0,
    "t35170" : 0,
    "t35171" : 0,
    "t35172" : 0,
    "t35173" : 0,
    "t35174" : 0,
    "t35175" : 0,
    "t35176" : 0,
    "t35177" : 0,
    "t35178" : 0,
    "t35179" : 0,
    "t35180" : 0,
    "t35181" : 0,
    "t35182" : 0,
    "t35183" : 0,
    "t35184" : 0,
    "t35185" : 0,
    "t35186" : 0
  },
  "startTime" : "2022-03-25T22:06:16.458Z"
}
DETAIL:  Key (project_id, group_id, task_id)=(-Myiz41plx7m5ya_JlCf, g1050, t34299) is not present in table "tasks".
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/mapswipe_workers/firebase_to_postgres/transfer_results.py", line 93, in transfer_results_for_project
    save_results_to_postgres(results_file)
  File "/usr/local/lib/python3.8/dist-packages/mapswipe_workers/firebase_to_postgres/transfer_results.py", line 292, in save_results_to_postgres
    p_con.query(query_insert_results)
  File "/usr/local/lib/python3.8/dist-packages/mapswipe_workers/auth.py", line 64, in query
    self._db_cur.execute(query, data)
psycopg2.errors.ForeignKeyViolation: insert or update on table "results" violates foreign key constraint "results_project_id_fkey2"
DETAIL:  Key (project_id, group_id, task_id)=(-Myiz41plx7m5ya_JlCf, g1050, t34299) is not present in table "tasks".

Hagellach37 avatar Mar 26 '22 18:03 Hagellach37