Bug: Provisioning of multiple pipelines in a yaml file fails if one of them has an invalid version

[nickchomey]

Bug description

I was exploring how the multiple pipelines per yaml config worked and was trying to make it fail on one of them.

When one has an invalid version, the whole provisioning fails with an error like ERR provisioning failed error="could not parse file \"/home/nick/conduittest/pipelines/test.yaml\": parsing error: unrecognized version 4"

Conversely, if you omit the connectors section from a pipeline, the others will provision and start and you just get this error ERR provisioning failed error="pipeline \"yaml-test-pipeline-3\", error while provisioning: could not start the pipeline \"yaml-test-pipeline-3\": could not build nodes for pipeline yaml-test-pipeline-3: can't build pipeline without any source connectors" for the bad pipeline.

Steps to reproduce

You could try using a config like this

---
version: 2.2
pipelines:
  - id: yaml-test-pipeline-1
    description: First pipeline created via YAML API
    connectors:
      - id: test-source-1
        type: source
        plugin: builtin:file
        settings:
          path: ./example.in
      - id: test-destination-1
        type: destination
        plugin: builtin:file
        settings:
          path: ./example-yaml-1.out
    processors: []
---
version: 2.2
pipelines:
  - id: yaml-test-pipeline-2
    description: Second pipeline created via YAML API
    connectors:
      - id: test-source-2
        type: source
        plugin: builtin:file
        settings:
          path: ./example.in
      - id: test-destination-2
        type: destination
        plugin: builtin:file
        settings:
          path: ./example-yaml-2.out
    processors: []
---
version: 4
pipelines:
  - id: yaml-test-pipeline-3
    description: Malformed pipeline with invalid version
    connectors:
      - id: test-source-3
        type: source
        plugin: builtin:file
        settings:
          path: ./example.in
      - id: test-destination-3
        type: destination
        plugin: builtin:file
        settings:
          path: ./example-yaml-3.out
    processors: []

Version

0.13.4

[lovromazgon]

Thanks for reporting this @nickchomey! Even though it's clearly a bug, I see it as pretty low-priority, as the workaround is straightforward - split the pipelines into separate configuration files. In that case, the valid files should load correctly. We're going to keep this open in case someone stumbles on the same problem, but we don't plan to invest time in this right now.

As always though - happy to review PRs that fix the behavior 😉

[nickchomey]

Yes, definitely not a high priority - I dont think I'll be contributing a PR for it. I just figured it was worth reporting/documenting.