Add status indication during the synchronization process in the Mobile app [5]

[SORMAS-ZohaibHassan]

Problem Description

When we try to synchronize in the mobile app, there is no indication that how many cases, contacts, tasks, etc need to be synchronized. There is just a loading icon that does not tell any information that how much data or timing is remaining. In this way, the end-user is just blindly waiting and does not know how much time it needs or how much data it needs to be synced yet.

Proposed Change

Add some sort of progress bar or a Sync Status tab where the user can see, when it was last synced, and if the synchronization is in progress then how much data is already synced and ~~how much is remaining~~, and ~~how much time is needed more based on current bandwidth~~. This type of information will also help us to improve the mobile app user experience and performance.

Details

The main idea is to:

1. Display the steps that are executed during a synchronization (sending new data, receiving infrastructure data, ...).
2. For each step display which entities are being synchronized and how many of them (only for pushed data and for data that has been pulled)

PS: The actual design could obviously be improved, but I'd leave this for later - first of all we should see whether it makes sense.

• [ ] When a synchronization is started, list the steps for the user, as listed below
• [ ] When data is pushed, also display how many entities are being pushed
• [ ] When data is pulled display how many entities already have been pulled during the current sync process
• [ ] Update the displayed data after every server roundtrip

The steps that are executed are different based on the three synchronization types:

sync changes

1. Send new data
2. Pull infrastructure data
3. Delete obsolete data (once per day)
4. Synchronize data
5. Pull modified data

Step 5 is needed, because pushing data may change other data that should be pulled afterwards. Not sure how to name this, so users aren't confused why data is pulled twice.

sync complete

1. Pull infrastructure data
2. Pull uuids of infrastructure to delete invalid and pull missing
3. Send new data
4. Pull uuids of all data to delete invalid and pull missing
5. Synchronize data
6. Pull modified data

sync complete and re-pull

1. Pull infrastructure data
2. Pull uuids of infrastructure to delete invalid and pull missing
3. Re-pull all data
4. Pull uuids of all data to delete invalid
5. Synchronize data
6. Pull modified data

Acceptance Criteria

• Users can see what type of data is currently being synchronized
• Users can see the counts for data that has been synchronized
• Synchronization does not leave the user in a situation where nothing is happening for 20 seconds or more.

Implementation Details

• [ ] AbstractAdoDao needs to provide count methods for queryForNew and queryForModified. Those are meant to provide the counts needed to display what is going to be pushed
• [ ] We need a representation of the steps (send, pull, ...) and sub-steps (pull persons, push persons, ...) that are done during synchronization. This representation is used to update the progress dialog. In a separate issue we may want to temporarily keep this information to allow users to see the progress of the last or current synchronization that was done
• [ ] The representation needs to be updated after every server communication during synchronization

Additional Information

The screenshot is attached for current implementation.


The dto helper classes should ideally be stateless, but currently have lastSyncedEntityDate + Uuid fields. This may have to be changed, because it's possibly negatively affecting the re-pull logic. Discuss with @BarnaBartha and decide whether an issue should be created for this.

Follow-up issues

1. Allow users to see details on the last or current (if done in background) sync process
2. Display actual numbers of entities to be pulled from the server
3. Calculate an approximate time for the whole sync process

[StefanKock]

Refinement needed by @MartinWahnschaffe and @MateStrysewske concerning what should be shown and how updated.

[MartinWahnschaffe]

Discussed this with @MateStrysewske and created a mockup on what it could look like. For this issue we would like to focus on displaying more details on what exactly is being synchronized, but not on how long it takes - this can go into a separate issue, once we have implemented this.

The main idea is to:

1. Display the steps that are executed during a synchronization (sending new data, receiving infrastructure data, ...).
2. For each step display which entities are being synchronized and how many of them.

Previous details. This has been refined and moved to the issue description above:

The steps that are executed are different based on the three synchronization types:

sync changes

1. push new data
2. pull infrastructure
3. delete obsolete data (once per day)
4. pull data
5. push data
6. pull changed data

Step 6 is needed, because pushing data may change other data that should be pulled afterwards. Not sure how to name this, so users aren't confused why data is pulled twice.

sync complete

1. pull infrastructure
2. pull uuids of infrastructure to delete invalid and pull missing
3. push new data
4. pull uuids of all data to delete invalid and pull missing
5. pull data
6. push data
7. pull changed data

Step 5 is not needed. The pull uuid logic needs to be re-evaluated in terms of performance and actual usefulness.

sync complete and re-pull

1. pull infrastructure
2. pull uuids of infrastructure to delete invalid and pull missing
3. re-pull all data
4. pull uuids of all data to delete invalid and pull missing
5. pull data
6. push data
7. pull changed data

Step 5 (and likely also step4) is not needed. The pull uuid logic needs to be re-evaluated in terms of performance and actual usefulness.

Implementation remarks

• when a synchronization is started, list the steps for the user
• for each step list the entities that are synchronized and the actual counts. I'd suggest to not display entities when the count is 0.
• when data is pushed/pulled update the counts accordingly
• for pushing data, we can calculate the total numbers on the local device
• for pulling data, we will have to do another server round-trip to get the total numbers based on the current change dates from the server
• pull + push of entities are currently done together for each entity type. This should be split, doing a pull for all types first and then the push
• the dto helper classes should ideally be stateless, but currently have lastSyncedEntityDate + Uuid fields. This may have to be changed, because it's possibly negatively affecting the re-pull logic. Discuss with @BarnaBartha and decide whether an issue should be created for this.

@zohaibhassan122 Please let us know what you think about the mockup and the general idea of displaying both the steps and the actual synchronized entities.

PS: The actual design could obviously be improved, but I'd leave this for later - first of all we should see whether it makes sense.

[SORMAS-ZohaibHassan]

Hi @MartinWahnschaffe, The mockup looks good to me. I have maybe two questions/concerns in general.

1. When synchronization is happening in the background, How the user will see this progress window?
2. As you mentioned in the implementation remarks that for pushing data, we will do a total count on the local device and for pushing data, we will do another round trip for total counts.

Do these extra calculations have an impact on performance? if yes. then what would be the scale?

I am trying to understand the overhead cost. Maybe when the amount of data is small then the overhead cost is negligible but not in the case of huge data amounts. In this scenario, maybe we can offer a enable/disable button for "Detailed Progress Window". Or it would not be possible to have such kind of option? What are your general thoughts on it?

[MartinWahnschaffe]

Thanks. We talked about it today in the refinement and decided to split getting the total numbers out into a separate issue. This means this issue will only displayed what data has been received and which entity types still need to be pulled, but not the number of entities to be pulled. This is a good first step anyway and splitting it out doesn't have a negative impact on the overall workload.

I will refine the issue description later today.

[MartinWahnschaffe]

I have refined the issue now. I had to update the mockup a bit, because I realized that we have to keep the current synchronization logic that does a pull + push at once for each entity type, instead of doing a pull for all entity types first and then a push for all entity types. The reason is that pushing data of one entity type (e.g. case) may modify data of other entity types (e.g. contacts) which would then lead to the sync not working for the updated data.

In an ideal solution we would only send the data fields that have actually been changed to the server which would enable us to do an partial update without running into conflicts.

@zohaibhassan122 Regarding your first question: I have defined this issue now in a way that makes it easy for us to hold the information of the current and previous sync process. This will allow us to display these information on demand. This needs to be defined in a separate issue though. Should I already create this issue or should we do this first and see how it works out, before defining additional follow-up work?

[SORMAS-ZohaibHassan]

Hi @MartinWahnschaffe, Thanks for the refinement. I think you can create the other issue later for follow-up work.

[MateStrysewske]

@MartinWahnschaffe has discovered a problem with the most recent changes, i.e. he gets an assertion error during the initial synchronization. As I can't reproduce this on my system right now, he will take over fixing this problem.

[adinaflorea9]

I am reopening the ticket so that the following can be investigated:

• [x] Infrastructure data seems to not be counted during the synchronization process in the status indication. Steps to reproduce:

1. In the web app, create a new continent, subcontinent, area, region, district, community and facility/point of entry;
2. Trigger either a manual synchronization in the mobile app, or a synchronization from the 'Re-synchronize data' button.

GIF of the behavior:

• [x] Contacts and immunizations edited in the mobile app seem to not be counted in the synchronization status. Steps to reproduce:

1. In the mobile app, edit a contact and/or an immunization;

GIF of the behavior: Contacts:

Immunizations:

[adinaflorea9]

I am reopening the ticket due to the following: when a Surveillance Officer is logged in the mobile app, tasks associated with any type of entity (event, case, contact) are counted in the synchronization status as 'Visits':

[adinaflorea9]

Verified ticket on the local machine using the latest version of SORMAS from the development branch - 1.76.0-SNAPSHOT(40b7f83).