at_client_sdk
at_client_sdk copied to clipboard
build client performance test rigs
**Is your feature request related to a problem? Please describe. Build the tests that measure the performance bottlenecks
**Describe the solution you'd like Use design from #733
**Describe alternatives you've considered NA
**Additional context NA
See comment for more reference
- [x] Notification Latency test As part of notification latency, Added tests that include
-
Sender, that sends a notification text (notify: text) continuously to a receiver without any expiry,
sender_test -
The first receiver that only receives(without sync), and receiver_test
-
a second receiver that listens to the notification from the sender and also updates a key for every 500ms i.e..,( with sync ) receiver2_test
Observations
The latency results were better for the receiver that only received notifications than the receiver that received notifications and updated a key simultaneously.
A graphical representation of the same can be found here.
https://docs.google.com/spreadsheets/d/1wY9eFI1OdqaexyR7Ho7ctT6IyNdsdIIB/edit#gid=63467969
Notification latency tests need to be re-tested once the following bug is fixed. https://github.com/atsign-foundation/at_client_sdk/issues/750
- [x] Sync Latency tests Sync latency tests were performed with different values for the following parameters syncRequestThreshold - The number of requests needed for a sync to be triggered. syncRequestTriggerInSeconds - The number of seconds after which a sync should be triggered syncRunIntervalSeconds - The interval after which the sync runs.
- Latency test with default values (3,3,5), without calling sync() manually and with a wait for 10 seconds.
- Latency test with default values(3,3,5) and by calling sync() in the test.
- Latency test with threshold values (1,3,3)
- Latency test with threshold values (2,3,3)
- Latency test with threshold values (1,3,2)
- Latency test with threshold values (1,2,1)
- Latency test with threshold values (1,2,2)
Test can be found in the following branch sync_latency_test
Observations The best latency time of around 1118ms on average, calculated for 50 iterations was for the following threshold values. syncRequestThreshold 1, syncRequestTriggerInSeconds 2 syncRunIntervalSeconds 1
Results for the above-mentioned latency scenarios can be found in the following sheet. https://docs.google.com/spreadsheets/d/13T4poZNMjIxSU2KM9EluaJ0B1TRiKXu-/edit#gid=1637466287
- [ ] Design isolates to better study the parallelism required in at_client
Re-tested with the skip_latestNotificationId. Now the latency results are better compared to the previous results. Results can be found in the following sheet.
Tab -1 Previous results(without the fix for skipping latestnotification Id) Tab 2 - After fix (Receiver that only listens to the notification) Tab -3 After fix (Receiver that listens to the notifications and updates the key )
https://docs.google.com/spreadsheets/d/1wY9eFI1OdqaexyR7Ho7ctT6IyNdsdIIB/edit#gid=1926778118
Latency tests have been re-verified again after the fix for "skipping latest notification Id ". The results have been better than the previous results that have been taken without the fix. Results can be found in the following sheet.
https://docs.google.com/spreadsheets/d/1wY9eFI1OdqaexyR7Ho7ctT6IyNdsdIIB/edit#gid=243961569
Tab 2 and 3
Moving to backlog as the core work in PR50 will be to continue to focus on sync tests, rearchitecting for secure element, future-proofing our encryption and of course bug fixing