dvc icon indicating copy to clipboard operation
dvc copied to clipboard

dvc.exceptions.CyclicGraphError: Pipeline has a cycle involving: load_extract_save.

Open Padmanabhan100 opened this issue 3 years ago • 4 comments

stages:
  load_extract_save: 
    cmd: python src/stage_01_load_extract_save.py --config=config/config.yaml
    deps:
      - config/config.yaml
      - src/utils/all_utils.py
      - src/stage_01_load_extract_save.py
      - artifacts/data
    outs:
      - artifacts/data
      - artifacts/clean_data/X.npy
      - artifacts/clean_data/Y.npy

  train_test_split_save:
    cmd: python src/stage_02_train_test_split_save.py --config=config/config.yaml --params=params.yaml
    deps:
      - artifacts/clean_data/X.npy
      - artifacts/clean_data/Y.npy
      - src/utils/all_utils.py
      - params.yaml
      - config/config.yaml
      - src/stage_02_train_test_split_save.py
    outs:
      - artifacts/train_data/X_train.npy
      - artifacts/train_data/Y_train.npy
      - artifacts/test_data/X_test.npy
      - artifacts/test_data/Y_test.npy
  
  train_model:
    cmd:  python src/stage_03_train.py --config=config/config.yaml --params=params.yaml
    deps:
      - artifacts/train_data/X_train.npy
      - artifacts/train_data/Y_train.npy
      - artifacts/test_data/X_test.npy
      - artifacts/test_data/Y_test.npy
      - src/stage_03_train.py
      - src/utils/all_utils.py
      - config/config.yaml
      - params.yaml
    outs:
      - artifacts/checkpoints
      - artifacts/model
  
  metrics:
    cmd: python src/stage_04_metrics.py --config=config/config.yaml
    deps:
      - src/stage_04_metrics.py
      - config/config.yaml
      - src/utils/all_utils.py
      - artifacts/test_data/X_test.npy
      - artifacts/test_data/Y_test.npy
      - artifacts/checkpoints
      - artifacts/model
    outs:
      - confusion_matrix.png

This is my dvc.yaml.

I have created github workflows, but the run fails and is giving me this error - 2022-06-17 21:34:59,843 ERROR: Pipeline has a cycle involving: load_extract_save.

I have tried down grading the dvc module but nothing works.

Padmanabhan100 avatar Jun 17 '22 21:06 Padmanabhan100

You have artifacts/data listed as both a dependency and an output in that stage, which is not supported in DVC (due to being a cyclic/circular dependency)

pmrowla avatar Jun 18 '22 01:06 pmrowla

Thanks a lot, it worked😁

Padmanabhan100 avatar Jun 18 '22 11:06 Padmanabhan100

Thanks for clarifying @pmrowla! It might be worth putting this clarification into the error message itself e.g. ERROR: same item(s) defined as both a dependency and an output in stage: <stage name>

alex000kim avatar Aug 08 '22 21:08 alex000kim

Reopening as a Good First Issue for the UI enhancement.

Would need to update

https://github.com/iterative/dvc/blob/f7fbac1eed12c8e5e7b6657fd741d4f426a6fe26/dvc/exceptions.py#L132-L138

Used in https://github.com/iterative/dvc/blob/main/dvc/repo/graph.py#L27

daavoo avatar Aug 09 '22 07:08 daavoo