Endpoint performance

[MichaelAkvo]

Endpoint performance has been a long-standing issue with some endpoints taking multiple seconds to complete.

This epic will track the most problematic endpoints and try to find either targeted or general improvements.

[MichaelAkvo]

After profiling the tests, some of the heaviest methods (excluding migrations) are:

• akvo.rest.viewsets._projects_filter_for_non_privileged_users
• akvo.rsr.models.model_querysets.project.ProjectQuerySet.of_partners
• akvo.rsr.models.user.User.my_projects

My initial assumption was that there were too many models, but it seems like our implementation of hierarchies and the permission system are the cause of the lethargy. The aforementioned methods along with other candidates make an inordinate amount of DB calls. Other candidates that look dangerous but aren't part of the tests (we don't have 100% coverage) are:

• db calls in while True loops:
  • akvo.rest.views.program_results_geo.fetch_contributors
  • akvo.rest.views.project_overview.get_contributor_ids
  • akvo.rest.views.project_overview._get_indicator_hierarchy_ids
  • akvo.rest.views.project_overview._get_indicator_periods_hierarchy_flatlist
  • akvo.rsr.project_overview.get_periods_hierarchy_flatlist
  • akvo.rsr.models.result.utils.get_dimension_value_hierarchy_flatlist
  • akvo.rsr.usecases.change_project_parent.get_lineage
  • akvo.rsr.views.py_reports.nuffic_country_level_map_report.get_period_hierarchy_flatlist
• db calls in other loops
  • akvo.rsr.models.result.result.Result.descendants
  • akvo.rsr.models.project.Project.descendants
  • akvo.rsr.admin.OrganisationAdmin.get_queryset
  • akvo.rsr.admin.ProjectAdmin.get_limited_queryset
  • akvo.rsr.admin.PartnerSiteAdmin.get_queryset
  • akvo.rsr.admin.EmploymentAdmin.get_queryset
  • akvo.rsr.project_overview.get_disaggregations
  • akvo.rsr.project_overview.UpdateCollection._build
  • akvo.rsr.models.employment.Employment.to_dict
• lack of batch CRUD (these are less critical)
  • akvo.rest.serializers.indicator_period_data.IndicatorPeriodDataFrameworkSerializer.create
  • akvo.rest.serializers.indicator_period_data.IndicatorPeriodDataFrameworkSerializer.update
  • akvo.rest.serializers.project_document.ProjectDocumentSerializer.create
  • akvo.rest.serializers.project_update.ProjectUpdateSerializer.create
  • akvo.rest.views.default_period.project_default_periods
  • akvo.rest.views.employment.change_user_roles
  • akvo.rest.views.indicator_period.bulk_add_periods
  • akvo.rest.views.indicator_period.bulk_remove_periods
  • akvo.rsr.views.project.main

Honarable mentions

akvo.rest.views.program_results_geo.ReportingPeriodMixin

This has fields that are lists of of other models, leading to an inability to make querysets. The lists are then filtered in other methods in order to generate querysets e.g:

• akvo.rest.views.program_results_geo.ContributorData.contributing_countries
• akvo.rest.views.program_results_geo.PeriodData.contributing_countries
• akvo.rest.views.program_results_geo.IndicatorData.contributing_countries
• akvo.rest.views.program_results_geo.ResultData.contributing_countries
• akvo.rest.views.program_results_geo.hierarchize_contributors

akvo.rsr.views.py_reports

These make use of views or viewproxies and have nested loops of querysets. I haven't tested them yet, but as far as I can see, little use is made of QuerySet.select_related(...) and QuerySet.prefetch_related(...).

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.