kestra icon indicating copy to clipboard operation
kestra copied to clipboard

GitHub Action and CLI Validate marking 1 flow as invalid when it's valid

Open japerry911 opened this issue 5 months ago • 16 comments

Describe the issue

Execute:

./run kestra:cli 1 flow validate kestra_flows/automations/bigquery --server=https://us.kestra.cloud --api-token=<API-TOKEN> --tenant=<TENANT-DEV-ID>

Result:

✘ - x.automations.bigquery.bigquery_dedupper
        Unable to parse flows due to the following error:
        - tasks[0].taskRunner.region: must not be null

Workflow YAML:

...
tasks:
  - id: run_deduplication_python_command
    type: io.kestra.plugin.scripts.python.Commands
    beforeCommands:
      - export PYTHONPATH=src
      - export GCP_PROJECT_ID={{ secret('GCP_PROJECT_ID') }}
    containerImage: <valid-image-uri>
    commands:
      - >
        python src/automations/bigquery/dedupper/main.py 
        ...
    warningOnStdErr: false
    taskRunner:
      type: io.kestra.plugin.ee.gcp.runner.Batch
      waitUntilCompletion: 18000
      machineType: e2-micro
      computeResource:
        cpu: "2000"
        memory: "1024"
...

Plugin Defaults in root namespace:

- type: io.kestra.plugin.scripts.python.Commands
  forced: false
  values:
    taskRunner:
      type: io.kestra.plugin.ee.gcp.runner.Batch
      bucket: "{{ secret('GCS_BUCKET') }}"
      projectId: "{{ secret('GCP_PROJECT_ID') }}"
      region: <gcp valid region string here>
      serviceAccount: "{{ secret('GCP_SA_JSON') }}"

I think it does not validate well when plugin_defaults is present for ee.batch region.

Let me know your all's thoughts, or if you need me to do anything debug-wise.

Thank you team 🙏 🚀

Environment

  • Kestra Version: 0.18.6
  • Operating System (OS/Docker/Kubernetes): Kestra Cloud (validate command ran locally in docker container with CLI command 0.18.6)
  • Java Version (if you don't run kestra in Docker): n/a

japerry911 avatar Sep 13 '24 21:09 japerry911