[S2S] Add a duplicate detection warning when sharing a case with another instance [5]

[AliciaRaber]

Problem Description

When sharing a case via S2S between one system (source system) and another system (target system) currently there’s no check in place if the case, contact or person that is about to be accepted already exists in the target system.

This could potentially lead to duplicates in the system and multiple entries for the same case being reported to RKI.

For S2S to work properly and sync changes to entities it is crucial for the entities to have the same UUID in both systems. Which means we can’t just simply use the same process for duplicate detection as we do when creating new entities through the UI (the user selects a case or person from a list and uses that one instead). This could potentially lead to the S2S connection breaking because the user can select an entity with a different UUID.

Proposed Change

When accepting a case that was send via S2S the target system needs to run the normal duplicate detection feature for the entities that are about to be accepted. If at least one potential duplicate has been found, instead of showing a list of potential duplicate candidates the system just shows a warning message, prompting the user to manually check for duplicates and merge them.

In order to prevent double input and double report to RKI, the target system has to check if case data is existing. Therefore, the duplicate detection mechanism have to check the data in target system. The problem is that the place of stay and/or responsible region from the source system is different to the data in target system. Therefore, in first place the target system has to check the person before suggesting an associated case. After indicating a person and associated case in target system it is necessary to inform the user that he is about to accept a case that is already in target system and it is possible to merge cases afterwards.

If case is shared with target system it is still possible to change person data in source and target system. Case data is not affected.

The merge has to be limited. In order to prevent double share after accepting the case in the target system the merge should be only possible for accepted shared case. Otherwise the person uuid and case uuid gets overwrite by non-shared case and it would be possible to revoke share and share case again with target system.

Checks need to be executed as follows (Point of view: target system)

• User accepts a share request from a source system
• Target system receives full data set from source system
• Entities are created inside target system
• Run duplicate check for case / contact (depending on what was shared)
• Save number of possible duplicates
• Run duplicate check for person
• Add number of possible duplicates to the previous number If the sum of possible duplicates found is greater than 0 Yes: Show warning message with appropriate message (see Add. Information) else No: No action necessary

Duplicate warning opens and shows following options:

• OK --> Closes dialog and case gets imported in the target system
• Cancel --> Closes dialog and case gets not imported in target system
• OK and continue to merge overview --> Closes dialog, case gets accept in target system and user gets routed to merging overview in order to merge cases according to just accepted case in target system.

After accepting the case in target system and potential merge of the case the person data can still be updated and gets synchronized in both system. The merged case has also to be marked as shared case (not even in the case overwiew, also in the search overview). There should be a filter setting for shared cases (even if they are merged afterwards). If person merge function is implement afterwards the synchronisation afterwards will not work anymore if person is merged in source or target system. This has to be applied and noted for developement of person merging.

Acceptance Criteria

Implementation Details

Additional Information

Warning messages in german:

• Kein Duplikat erkannt.
• In Ihrem System befindet sich bereits eine Information zur Person. Die Person wird durch das Aufnehmen nicht in einen Fall konvertiert.
• In Ihrem System befindet sich bereits ein Kontakt zu dieser Person.
• In Ihrem System befindet sich bereits ein Fall zu dieser Person. Sollten Sie die Anfrage annehmen, ist der Fall als Duplikat in Ihrem System. Sie sollten nach der Annahme die Fälle konsolidieren. Bitte achten Sie darauf, dass Sie dazu den angenommen Fall verwenden. Andernfalls kann es zu einer erneute Übermittlung führen.
• In Ihrem System befindet sich bereits ein Kontakt und Fall zu dieser Person. Sollten Sie den Fall annehmen, ist die Konvertierung des Kontaktes nicht mehr notwendig, sofern Sie die Konvertierung schon vorgenommen haben.
• In Ihrem System befindet sich bereits eine Person und ein Kontakt mit dem gleichen Namen. Nachdem Sie den Fall aufgenommen haben, ist es unter Umständen notwendig, eine manuelle Konvertierung des Kontakts zu dem angenommenen Fall vorzunehmen. Sie sollten nach der Annahme die Fälle konsolidieren. Bitte achten Sie darauf, dass Sie dazu den angenommen Fall verwenden. Andernfalls kann es zu einer erneute Übermittlung führen. Nachdem Sie den Fall aufgenommen haben, ist es unter Umständen notwendig, eine manuelle Konvertierung des Kontakts zu dem angenommenen Fall vorzunehmen.

Flow diagram scope in target system:

Mockups:

Mockups here

[leventegal-she]

After accepting the case in target system and potential merge of the case the person data can still be updated and gets synchronized in both system. The merged case has also to be marked as shared case (not even in the case overwiew, also in the search overview).

This would mean that we have to change the merge functionality and alway keep the shared case as lead case.

If person merge function is implement afterwards the synchronisation afterwards will not work anymore if person is merged in source or target system. This has to be applied and noted for developement of person merging.

I don't understand this. Currently if the not shared case is picked no merge is done the synchronization will not work anymore.

[markusmann-vg]

@AliciaRaber We don't understand one of your warning message

1. "In Ihrem System befindet sich bereits eine Information zur Person. Die Person wird durch das Aufnehmen nicht in einen Fall konvertiert." => Persons are not shared directly and so are not converted to cases
2. Why should we add a warning message when no duplicate was found? Can't we ignore that case or is this explicitly wanted?

Side note: There is no person merge yet

This would be our implementation summary:

Duplicate Detection is triggered, when the user is about to accept a share request (clicks for accepting)

When the duplicate detection finds at least one duplicate, then a warning message is shown, that asks the user to manually check for duplicates and merge when wanted

Side Note: Obviously merging can only be done with accepted shared cases

Duplicate warning opens and shows the 3 following buttons below the actual warning message:

• „OK“ 
Clicking this: Closes the dialog and case gets imported in the target system
• „Cancel“
Clicking this: Closes the dialog and case does not get imported in target system
• „OK and continue to merge overview“ 
Clicking this: Closes the dialog, case gets accept in target system and user gets routed to merging overview in order to merge cases according to just accepted case in target system.

The warning message can differ, depending on the result of the duplicate check:

• There is already a contact for this person in your system.
• There is already a case for this person in your system. If you accept the request the case will be in your system as a duplicate.
You should consolidate the cases after accepting. Please make sure that you use the accepted case for this purpose ~~* There is already a contact and case for this person in your system. If you accept the case or contact, the conversion of the contact is no longer necessary, if you have already done the conversion.~~

~~* There is already a person and a contact with the same name in your system. After you have accepted the case, it may be necessary to perform a manual conversion of the contact to the accepted case.
You should consolidate the cases after acceptance. Please make sure that you use the accepted case for this purpose. Otherwise, it may result in retransmission.
After you have accepted the case, it may be necessary to perform a manual conversion of the contact to the accepted case.~~

[leventegal-she]

Identified the following situations:

Accepting case

1. There is at least one similar case:

• Message: There is at least one similar case in your system. If you accept the request the case will be in your system as a duplicate. You should consolidate the cases after accepting. Please make sure that you use the accepted case for this purpose.

2. From the similar cases at least one similar case that was converted from a contact

• Message: There is at least one similar case in your system. If you accept the request the case will be in your system as a duplicate. You should consolidate the cases after accepting. After consolidating the cases, it may be necessary to perform a manual conversion of some contacts to the accepted case. Please make sure that you use the accepted case for this purpose.

3. There is a similar contact not converted to case

• Message: There is at least one similar contact in your system. After accepting the request, it may be necessary to perform a manual conversion of those contacts to the accepted case
• Option to Go to contact directory instead of Go to merge overview.

Accepting contact

1. There is at least one similar contact

• Message: There is at least one similar contact in your system. If you accept the request the contact will be in your system as a duplicate. You should consolidate the contacts after accepting. Please make sure that you use the accepted contact for this purpose.

2. From similar contacts at least one is converted to case

• Message: There is at least one similar contact in your system. If you accept the request the contact will be in your system as a duplicate. You should consolidate the contacts after accepting. After consolidating the contacts, it may be necessary to perform a manual conversion of the accepted contact. Please make sure that you use the accepted contact for this purpose.

3. There is at least one similar case

• Message: There is at least one similar case in your system. After accepting the request, it may be necessary to perform a manual conversion of the accepted contacts.
• Option to Go to contact instead of Go to merge overview.

Accepting case or contact

There are only similar persons (no similar case and/or contact)

• Message: There is at least one similar person in your system. If you accept the request the case/contact maybe will create a person in your system as a duplicate. Please check this after accepting.
• Option to Go to persons instead of Go to merge overview.

[SahaLinaPrueger]

Thank you for implement it for contacts as well. Would be nice, if we can phrase more carefully: If you accept the request the case maybe will be in your system as a duplicate. If so, you should consolidate the cases after accepting.

In my opinion, the following error message is missing:

There is at least one similar person in your system. If you accept the request the case/contact maybe will create a person in your system as a duplicate. You should check this after accepting. Maybe you have to create a dummy case/contact in the 'old' person and merge the accepted case/contact with the dummy

[leventegal-she]

There is at least one similar person in your system. If you accept the request the case/contact maybe will create a person in your system as a duplicate. You should check this after accepting. Maybe you have to create a dummy case/contact in the 'old' person and merge the accepted case/contact with the dummy

@SahaLinaPrueger I'm not sure this is a valid use case, if only the person is duplicate it always comes with a case/contact so I don't see why to create another one in the system just to merge with the incoming one but at the and the user remains the with tho one coming through s2s. I see it as an unnecessary step. Or, does case/contact merge do also a person merge and the end result will be a single person with multiple cases? I don't know this kind of functionality.

I think this situation cannot be handled without a proper person merge functionality

[SahaLinaPrueger]

@leventegal-she Duplicate Detection Mechanism for cases and contacts always have a limit of 30 days. We need to find all possible duplicates. That is why we need a duplicate detection mechanism for persons, too.

Or, does case/contact merge do also a person merge and the end result will be a single person with multiple cases? I don't know this kind of functionality

If you merge two cases only one person remains. I just tested it again on 1.75.3 with salutation, email and phone number on the one person and occupation type and another phone number on the other person and they get merged into one person with salutation, occupation type, 2 phone numbers and one e-mail. I also do not know the exactly rules how the system has to behave but some kind of person merge is happening.

[leventegal-she]

OK, good to know.

I'll add the message, and also update my previous comment with the identified situations.

Then we need another ticket for improving the user experience like:

Accepting case - point 3.

Option to Go to contact directory instead of Go to merge overview.

The directory should be pre-filtered to show only the similar contacts

Accepting case/contact and there is only person duplicate

Option to Go to persons directory instead of Go to merge overview.

The directory should be pre-filtered to show only the similar persons

For the other cases The merge overview to be pre-filtered to show only the accepted cases/contacts

@SahaLinaPrueger Is it ok to do these in a separate ticket?

[SahaLinaPrueger]

@leventegal-she thank you very much! If it is in a separate ticket would the ticket also be done in 1.76? And the separate ticket would only be about the pre-filter or also about the "go to contact/ person directory"?

[leventegal-she]

I think the new ticket would go to the next release because it's not an easy task and I would not block 1.76 @markusmann-vg

It would be only about pre-filter

[SahaLinaPrueger]

On the quick I do not see that a pre-filter is specified, so thanks for the idea and can be in the next sprint, or am I wrong here @AliciaRaber ?

[SahaLinaPrueger]

@leventegal-she I tested the 'person merge' again. The situation is the following:

1. Two persons. Each person has only one case as entity. If you merge this two cases person merge happens.
2. Two persons. One person has only one case as entity. The other person has two cases as entity: -> If the leading case is the case with more than one cases on his person: person merge happens -> If the leading case is the case with only one case on his person: person merge does not happen

If you create a dummy on the 'old person' there are more than two entities and leading case has to be always the shared case. So the message if the person is similar should only be: There is at least one similar person in your system. If you accept the request the case/contact maybe will create a person in your system as a duplicate. Please check this after accepting.

I am so sorry, can you just delete the following sentence, please?: You should check this after accepting. Maybe you have to create a dummy case/contact in the 'old' person and merge the accepted case/contact with the dummy.

[leventegal-she]

Created https://github.com/hzi-braunschweig/SORMAS-Project/issues/10569 for filtering directories when the user chooses accept and navigate

[leventegal-she]

Reopen because data sent with "Exclude personal data" should not go through duplicate detection because they don't contain enough relevant data to do a proper duplicate check.

Also changing the responsible jurisdiction is broken.

[SahaLinaPrueger]

@leventegal-she we updated the german translation, because the translation was a little bit confusing. If possible it would be nice if you can update the translations here, too.

[dragosb-vg]

Validated on de1-de3 test environments on Version: 1.76.0-SNAPSHOT (f34a7c8)

• Correct "Potential duplicates detected" pop-up messages are displayed when sharing cases/contacts.
• Sharing with "exclude personal data does not trigger the duplicate detection
• Action (Okay, Cancel) and redirect buttons ( to Person directory /Merge Overview) trigger the correct action/redirect. Below are some samples.