android-fhir icon indicating copy to clipboard operation
android-fhir copied to clipboard

Adds Benchmark module to evaluate the SDK's first run performance

Open vitorpamplona opened this issue 2 years ago • 2 comments

Fixes #1474, #825 and #1378, potential merger with #1414

Description Adds a simple benchmark module to run performance tests on the engine and workflow. The module is focused on the first-time runs, which drive most of the performance issues we are having. The tests must be run in order.

I am not convinced that yet another module needs to be added, but given our struggles with performance, it might be a good short-term investment.

Current results in nanoseconds (seconds,milli,micro,nano) on an older Moto Z Play XT1635-02: (Android8, 3GB RAM, Octa-core 2.0 GHz) with Geekbench: 147 single core and 544 multi-core. These are single runs, but first-time runs (no caching):

   49,577,870   ns    trace    AJacksonMapperBenchmark.loadJsonMapper
2,258,560,068   ns    trace    BFhirContextLoaderBenchmark.loadR4
2,642,449,743   ns    trace    BFhirContextLoaderBenchmark.loadR5
1,665,396,000   ns    trace    BFhirContextLoaderBenchmark.loadDstu3
6,920,873,294   ns    trace    CCqlEngineFhirContextLoaderBenchmark.loadDstu2FhirModelResolver
3,897,679,346   ns    trace    CCqlEngineFhirContextLoaderBenchmark.loadR4FhirModelResolver
2,907,996,227   ns    trace    CCqlEngineFhirContextLoaderBenchmark.loadDstu3FhirModelResolver
  630,341,938   ns    trace    DFhirJsonParserBenchmark.parseLightFhirLibrary
1,258,804,813   ns    trace    DFhirJsonParserBenchmark.parseLightFhirBundle
2,422,628,522   ns    trace    EElmJsonLibraryLoaderBenchmark.parseImmunicyCheckCqlFromFhirLibrary
2,481,479,986   ns    trace    EElmJsonLibraryLoaderBenchmark.parseFhirHelpersCqlFromFhirLibrary
  116,937,355   ns    trace    FEngineDatabaseBenchmark.createAndGet
8,807,674,783   ns    trace    GCqlEvaluatorBenchmark.evaluatesLibrary (evaluation only)

Pixel 6 GB7N6 (Android 12, 8GB RAM, Octa-core (2x2.80 GHz Cortex-X1 & 2x2.25 GHz Cortex-A76 & 4x1.80 GHz Cortex-A55) with Geekbench: 1036 single core and 2720 multi-core:

   21,242,350   ns    trace    AJacksonMapperBenchmark.loadJsonMapper
  335,387,573   ns    trace    BFhirContextLoaderBenchmark.loadR4
  356,097,657   ns    trace    BFhirContextLoaderBenchmark.loadR5
  271,354,411   ns    trace    BFhirContextLoaderBenchmark.loadDstu3
  693,789,348   ns    trace    CCqlEngineFhirContextLoaderBenchmark.loadDstu2FhirModelResolver
  500,146,485   ns    trace    CCqlEngineFhirContextLoaderBenchmark.loadR4FhirModelResolver
  366,984,538   ns    trace    CCqlEngineFhirContextLoaderBenchmark.loadDstu3FhirModelResolver
  152,651,122   ns    trace    DFhirJsonParserBenchmark.parseLightFhirLibrary
  208,492,962   ns    trace    DFhirJsonParserBenchmark.parseLightFhirBundle
  297,588,461   ns    trace    EElmJsonLibraryLoaderBenchmark.parseImmunicyCheckCqlFromFhirLibrary
  276,356,160   ns    trace    EElmJsonLibraryLoaderBenchmark.parseFhirHelpersCqlFromFhirLibrary
   32,813,435   ns    trace    FEngineDatabaseBenchmark.createAndGet
1,018,112,914   ns    trace    GCqlEvaluatorBenchmark.evaluatesLibrary (evaluation only)

Alternative(s) considered I tried building an app and a macro module, but I think this micromodule is a better approach.

Type Testing

Screenshots (if applicable)

Checklist

  • [x] I have read and acknowledged the Code of conduct.
  • [x] I have read the Contributing page.
  • [x] I have signed the Google Individual CLA, or I am covered by my company's Corporate CLA.
  • [x] I have discussed my proposed solution with code owners in the linked issue(s) and we have agreed upon the general approach.
  • [x] I have run ./gradlew spotlessApply and ./gradlew spotlessCheck to check my code follows the style guide of this project.
  • [x] I have run ./gradlew check and ./gradlew connectedCheck to test my changes locally.
  • [x] I have built and run the demo app(s) to verify my change fixes the issue and/or does not break the demo app(s).

vitorpamplona avatar Jul 06 '22 19:07 vitorpamplona

Codecov Report

Merging #1489 (d042214) into master (5f8790d) will increase coverage by 42.34%. The diff coverage is n/a.

@@              Coverage Diff              @@
##             master    #1489       +/-   ##
=============================================
+ Coverage          0   42.34%   +42.34%     
- Complexity        0      384      +384     
=============================================
  Files             0      150      +150     
  Lines             0     5405     +5405     
  Branches          0      971      +971     
=============================================
+ Hits              0     2289     +2289     
- Misses            0     2758     +2758     
- Partials          0      358      +358     
Impacted Files Coverage Δ
.../android/fhir/sync/upload/LocalChangesPaginator.kt 0.00% <0.00%> (ø)
...alidation/PrimitiveTypeAnswerMaxLengthValidator.kt 22.22% <0.00%> (ø)
...d/fhir/datacapture/utilities/TypeConversionUtil.kt 0.00% <0.00%> (ø)
...droid/fhir/db/impl/entities/PositionIndexEntity.kt 0.00% <0.00%> (ø)
...gle/android/fhir/db/impl/SQLCipherSupportHelper.kt 71.42% <0.00%> (ø)
...android/fhir/datacapture/QuestionnaireViewModel.kt 47.55% <0.00%> (ø)
...a/com/google/android/fhir/index/ResourceIndexer.kt 49.75% <0.00%> (ø)
...oid/fhir/workflow/FhirEngineTerminologyProvider.kt 0.00% <0.00%> (ø)
...r/datacapture/views/QuestionnaireItemHeaderView.kt 82.35% <0.00%> (ø)
...apture/views/QuestionnaireItemViewHolderFactory.kt 100.00% <0.00%> (ø)
... and 140 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

codecov[bot] avatar Jul 06 '22 21:07 codecov[bot]

Might close or enhance #1492 @Tarun-Bhardwaj

vitorpamplona avatar Jul 25 '22 14:07 vitorpamplona

@vitorpamplona Do you have any ETA for the changes?

aditya-07 avatar Sep 01 '22 13:09 aditya-07