at_client_sdk icon indicating copy to clipboard operation
at_client_sdk copied to clipboard

build client performance test rigs

Open VJag opened this issue 2 years ago • 4 comments

**Is your feature request related to a problem? Please describe. Build the tests that measure the performance bottlenecks

**Describe the solution you'd like Use design from #733

**Describe alternatives you've considered NA

**Additional context NA

VJag avatar Oct 03 '22 09:10 VJag

See comment for more reference

  • [x] Notification Latency test As part of notification latency, Added tests that include
  1. Sender, that sends a notification text (notify: text) continuously to a receiver without any expiry,
    sender_test

  2. The first receiver that only receives(without sync), and receiver_test

  3. a second receiver that listens to the notification from the sender and also updates a key for every 500ms i.e..,( with sync ) receiver2_test

Observations

The latency results were better for the receiver that only received notifications than the receiver that received notifications and updated a key simultaneously.

A graphical representation of the same can be found here.

https://docs.google.com/spreadsheets/d/1wY9eFI1OdqaexyR7Ho7ctT6IyNdsdIIB/edit#gid=63467969

Notification latency tests need to be re-tested once the following bug is fixed. https://github.com/atsign-foundation/at_client_sdk/issues/750

  • [x] Sync Latency tests Sync latency tests were performed with different values for the following parameters syncRequestThreshold - The number of requests needed for a sync to be triggered. syncRequestTriggerInSeconds - The number of seconds after which a sync should be triggered syncRunIntervalSeconds - The interval after which the sync runs.
  1. Latency test with default values (3,3,5), without calling sync() manually and with a wait for 10 seconds.
  2. Latency test with default values(3,3,5) and by calling sync() in the test.
  3. Latency test with threshold values (1,3,3)
  4. Latency test with threshold values (2,3,3)
  5. Latency test with threshold values (1,3,2)
  6. Latency test with threshold values (1,2,1)
  7. Latency test with threshold values (1,2,2)

Test can be found in the following branch sync_latency_test

Observations The best latency time of around 1118ms on average, calculated for 50 iterations was for the following threshold values. syncRequestThreshold 1, syncRequestTriggerInSeconds 2 syncRunIntervalSeconds 1

Results for the above-mentioned latency scenarios can be found in the following sheet. https://docs.google.com/spreadsheets/d/13T4poZNMjIxSU2KM9EluaJ0B1TRiKXu-/edit#gid=1637466287

  • [ ] Design isolates to better study the parallelism required in at_client

purnimavenkatasubbu avatar Oct 17 '22 06:10 purnimavenkatasubbu

Re-tested with the skip_latestNotificationId. Now the latency results are better compared to the previous results. Results can be found in the following sheet.

Tab -1 Previous results(without the fix for skipping latestnotification Id) Tab 2 - After fix (Receiver that only listens to the notification) Tab -3 After fix (Receiver that listens to the notifications and updates the key )

https://docs.google.com/spreadsheets/d/1wY9eFI1OdqaexyR7Ho7ctT6IyNdsdIIB/edit#gid=1926778118

purnimavenkatasubbu avatar Oct 20 '22 10:10 purnimavenkatasubbu

Latency tests have been re-verified again after the fix for "skipping latest notification Id ". The results have been better than the previous results that have been taken without the fix. Results can be found in the following sheet.

https://docs.google.com/spreadsheets/d/1wY9eFI1OdqaexyR7Ho7ctT6IyNdsdIIB/edit#gid=243961569

Tab 2 and 3

purnimavenkatasubbu avatar Oct 31 '22 05:10 purnimavenkatasubbu

Moving to backlog as the core work in PR50 will be to continue to focus on sync tests, rearchitecting for secure element, future-proofing our encryption and of course bug fixing

gkc avatar Nov 13 '22 16:11 gkc