Automatic deletion of personal data

[StefanKock]

Situation Description & Motivation

Data collected for cases, contacts and the other core entities has to be deleted automatically after a given time period.

Use cases

• Travel entries have to be deleted 14 days after the date of arrival in country
• Data collected within a case has to be deleted depending on different regulations. Some fields are kept for 10 years, others for only 3 years after the end of processing.

High-Level Explanation

We need to define and implement the automatic deletion of person-related data within SORMAS.

• The deletion has to be done privacy-compliant - there is no way to undo the deletion. (at least after a given time period)
• The given time period for the deletion should be given in SORMAS and can be modifyed by the Gesundheitsamt.
• Deletion is happening
• The Deletion concept can be found here (Important files: Main document; Annex 1(Anlage 01 - Löschregeln) ; Annex 4 (Anlage 04 - Umsetzungsvorgaben für einzelne IT-Systeme)): https://confluence.sormas-tools.de/pages/viewpage.action?pageId=14488475

Timeline

Core functionality should be done end of Q1 2022.

Tasks

Concept Phase

• [x] Specification of deletion concept & implementation work

Automatic deletion of entities

Note: There has been a change to the way this works. The first version of automatic deletion just does a soft-delete and then relies on mechanics that permantly delete soft-deleted data. Instead of this the automatic should now permanantly delete data, so manually deleted data can be handled separately.

• [x] #7008
• [x] #7240
• [x] #7246
• [x] #7845
• [x] #7247
• [x] #7319
• [x] #7249
• [ ] #8727
• [x] #8293
  • [x] #8909
• [x] #8295
• [ ] #8996
• [x] #9281
• [ ] #9666

Manual deletion of entities

• [x] #8162
• [ ] #8990
• [ ] #8465

Permanent deletion of entities

• [x] #7712
• [x] #7857
• [x] #7942
• [x] #7932
• [x] #7930
• [x] #7931
• [ ] #8010
• [x] #7713
• [x] #8343
• [x] #8936
• [x] #9061
• [x] #9094

Automatic archiving

• [x] #7775
• [x] #7774
• [x] #8595

Field-wise deletion

• [ ] #7760
• [ ] #7716
• [ ] #7762
• [ ] #7777

Needed Refinements

• [ ] Field-wise deletion rules for person and visit (both can be referenced by multiple different core-entities)

Out of scope

• Deletion of aggregated data like AggregateReport and WeeklyReport and also of infrastructure data like Region and Country is not handled by this epic
• Event Groups are an administrative entity and don't need to be automatically deleted
• Deletion of entites that were shared via S2S is currently not unregistering the system from receiving updates
• Deletion of entites can break the update chain for S2S, e.g. when a case is shared from A to B to C and then deleted in B, it will mean that C no longer receives updates from A via B.

Risks

Additional Information

• Archiving
• Manual & Automatic Deletion

[marko-arn]

I would like to point out that the handling should be different upon the case classification. A "No Case" might be earlier deleted then a confirmed case.

For the travel entries please also include the contacts which are only a travel entry and not a contact to a case or converted into a case.

Please keep also contacts which results in a case anyway.

[MartinWahnschaffe]

@Marko-ilmkreis Thanks for the input on "no case".

Could you give some more details on the travel entries + contacts point, please? Are you saying that you are creating a contact for a person that does have a travel entry, so the contact is created without an associated case and has the "returningTraveler" info set to true? Thus the contact would fall under the 14 day deletion period?

Would be good to have some details on this, so we can approach the data protection team.

[marko-arn]

@MartinWahnschaffe Yes, as the import of the DEA travel entries is not really possible before 1.67 we use the contacts and set the returningTraveler to true. Thus contacts must fall under the deletion after 14 days if they are not also a contact to a case, a ep or if they are not converted into a case. From 1.67 we hopefully will use the normal travel entries.

[MartinWahnschaffe]

Ok. I guess we will need to provide the means to manually delete those contacts in bulk, based on filters in the contact directory. The logic behind it will be a bit too specific to this workarround to extend the automatic deletion based on it.

[marko-arn]

The filter for travel returns is still there, so a filter for "not resulting in a case" and "not an ep" is missing and a filter by the date the entity was latest edited.

May be (offtopic) if would be easier to have a way to convert all "travel-entry-contacts" into travel entries and remove the button for travel return from contacts if the travel entries directory is enabled.

[MartinWahnschaffe]

I unfortunately didn't define a clear path on how to deal with campaigns. They are a core entity and can be manually deleted and as such they now also support automatic deletion, although this was not specified. In addition the permanent deletion of campaigns is not fully implemented, so it only works for campaigns that don't have campaign data yet.

There are now two ways forward:

1. Explicitly exclude campaigns form automatic and permanent deletion.
2. Extend the permanent deletion of campaigns to also include campaign data and extend the automatic deletion date checks to also include the dates of campaign data.

Solution 2 would be cleaner and more powerful, but comes with the drawback that deleting a whole campaign including all data may be a very big thing (comparable to deleting the aggregated reports of a whole country) that should not happen automatically.

Note: Campaigns are not used in Germany and thus aren't relevant for the related data security endeavor.

@Candice-Louw @Jan-Boehme Thoughts on this?

[Candice-Louw]

@SORMAS-JanBoehme - it may make sense to go with the option 1. Explicitly exclude campaigns form automatic and permanent deletion.

Reasoning: Campaigns only capture aggregate data, no personal/sensitive data (@MartinWahnschaffe - please correct this if I'm mistaken?) and as mentioned are not used in Germany. It therefore makes sense to exclude campaigns from the automatic and permanent deletion consideration based on these grounds.

[MartinWahnschaffe]

@Candice-Louw Yes and no. The intention of campaigns is to capture aggregate data. Campaign forms have text fields, though, so we can't say for sure whether it is miss-used for collecting personal/sensitive data.

[SORMAS-JanBoehme]

@Candice-Louw @MartinWahnschaffe I also think that automatically deleting the whole campaign and relevant data for analytics and decision making just to be sure that no one has entered personal / sensitive data in any text fields is a bit much.

The middle ground in my opinion would be to exclude campaigns that have data associated with them from the automatic deletion and add the standard mouse over info to text fields in which the users are informed about the GDPR agreement and that they must not enter any sensitive data in these fields like we have it in other forms