determined icon indicating copy to clipboard operation
determined copied to clipboard

perf: resolve get_experiment_checkpoints slowness

Open trentwatt opened this issue 2 years ago • 2 comments

Description

DET-8455

current get checkpoints for experiment does

SELECT *
FROM proto_checkpoints_view c
WHERE CAST(c.training->>'experiment_id' AS integer) = $1
ORDER BY c.report_time DESC

proto_checkpoints_view looks like

 SELECT c.uuid::text AS uuid,
    ...
    jsonb_build_object('trial_id', c.trial_id, 'experiment_id' ... )
   FROM checkpoints_view c; 

burying the experiment_id within the json_build_object prevents query planner from filtering by experiment_id before doing the join here:

   FROM checkpoints_v2 c
     LEFT JOIN trials t ON c.task_id = t.task_id
     LEFT JOIN experiments e ON t.experiment_id = e.id

resulting in scan of checkpoints.

adding c.experiment_id to proto_checkpoints_view and changing get_checkpoints_for_experiment.sql to

SELECT *
FROM proto_checkpoints_view c
WHERE c.experiment_id = $1
ORDER BY c.report_time DESC; 

resolves the query performance issues but breaks the proto conversion with

"error fetching checkpoints for experiment 48 from database: error running query: get_checkpoints_for_experiment: error converting row to Protobuf struct: proto: (line 1:62): unknown field \"experiment_id\""

(also breaks the other APIs that use this view).

Test Plan

gcloud sql connect determined-release-party

run the queries.

Commentary (optional)

further discussion in code comments

Checklist

  • [ ] Changes have been manually QA'd
  • [ ] User-facing API changes need the "User-facing API Change" label.
  • [ ] Release notes should be added as a separate file under docs/release-notes/. See Release Note for details.
  • [ ] Licenses should be included for new code which was copied and/or modified from any external code.
  • [ ] If modifying /webui/react/src/shared/ verify make -C webui/react test-shared passes.

trentwatt avatar Sep 22 '22 08:09 trentwatt

Deploy Preview for storybook-det canceled.

Name Link
Latest commit 350b9689f93f3f603dc1b1c3add3820b1170e81c
Latest deploy log https://app.netlify.com/sites/storybook-det/deploys/633b57101a8dec00097349d3

netlify[bot] avatar Sep 22 '22 08:09 netlify[bot]

Deploy Preview for determined-ui canceled.

Name Link
Latest commit 350b9689f93f3f603dc1b1c3add3820b1170e81c
Latest deploy log https://app.netlify.com/sites/determined-ui/deploys/633b571098598b00089c10e8

netlify[bot] avatar Sep 22 '22 08:09 netlify[bot]