determined icon indicating copy to clipboard operation
determined copied to clipboard

feat: add integration config for pachyderm input datasets (ET-12)

Open corban-beaird opened this issue 11 months ago • 2 comments

Description

Originally: Adds GET endpoint for URL that points to the pachyderm input repository (if configured) associated with provided experiment. This will be used to enable basic lineage views.

Adjusted: Instead of created a redundant endpoint, we're instead formalizing the expected fields in the pachyderm integration in expconf and making it accessible through existing API calls (SearchExperiments, GetExperiment, GetRun).

Experiment Config Addition:

integration:
    pachyderm:
        dataset:
            *project
            *repo
            commit
            branch
            *token  << this is a pachd token; however, since it's scoped to the live of the experiment, paychderm preferred to be placed under dataset 
        proxy:
            *host
            *port
            *scheme
        pachd:
            *host
            *port

Test Plan

  • Updated expconf tests to include the pachyderm integration config

Commentary (optional)

Checklist

  • [ ] Changes have been manually QA'd
  • [ ] User-facing API changes need the "User-facing API Change" label.
  • [ ] Release notes should be added as a separate file under docs/release-notes/. See Release Note for details.
  • [ ] Licenses should be included for new code which was copied and/or modified from any external code.

Ticket

ET-12

corban-beaird avatar Mar 01 '24 20:03 corban-beaird

Deploy Preview for determined-ui canceled.

Name Link
Latest commit afee2ae76de6950f9e40a54d1d9fefed4877b3ca
Latest deploy log https://app.netlify.com/sites/determined-ui/deploys/663a7739deb2d00008f611ec

netlify[bot] avatar Mar 01 '24 20:03 netlify[bot]

Codecov Report

Attention: Patch coverage is 14.43299% with 83 lines in your changes are missing coverage. Please review.

Project coverage is 44.58%. Comparing base (5b73dc4) to head (afee2ae). Report is 3 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8933      +/-   ##
==========================================
- Coverage   44.60%   44.58%   -0.03%     
==========================================
  Files        1275     1275              
  Lines      156385   156479      +94     
  Branches     2450     2449       -1     
==========================================
+ Hits        69756    69766      +10     
- Misses      86389    86473      +84     
  Partials      240      240              
Flag Coverage Δ
backend 41.76% <7.22%> (-0.07%) :arrow_down:
harness 64.09% <57.14%> (-0.01%) :arrow_down:
web 34.99% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
master/internal/api_experiment.go 56.79% <100.00%> (+0.04%) :arrow_up:
master/internal/api_runs.go 73.66% <100.00%> (+0.05%) :arrow_up:
master/internal/config/config.go 70.54% <ø> (ø)
master/pkg/schemas/expconf/experiment_config.go 64.21% <ø> (ø)
...r/pkg/schemas/expconf/zgen_experiment_config_v0.go 1.94% <0.00%> (-0.04%) :arrow_down:
harness/determined/common/api/bindings.py 40.21% <57.14%> (+0.02%) :arrow_up:
master/pkg/schemas/zgen_schemas.go 1.13% <0.00%> (-0.10%) :arrow_down:

... and 3 files with indirect coverage changes

codecov[bot] avatar Mar 18 '24 16:03 codecov[bot]