How to use multiple --filters for datafactory pipeline-run query-by-factory

[hwwilliams]

Describe the bug

Documentation mentions that --filters can be used multiple times but does not describe how.

Related command

az datafactory pipeline-run query-by-factory

Errors

None of these commands work.

az datafactory pipeline-run query-by-factory `
    --factory-name $DataFactoryName `
    --resource-group $ResourceGroupName `
    --filters operand="Status" operator="Equals" values="InProgress" `
    --filters operand="Status" operator="Equals" values="Queued" `
    --last-updated-after ((Get-Date).AddDays(-1) | Get-Date -Format "o" -AsUTC) `
    --last-updated-before (Get-Date -Format "o" -AsUTC)

az datafactory pipeline-run query-by-factory `
    --factory-name $DataFactoryName `
    --resource-group $ResourceGroupName `
    --filters operand="Status" operator="Equals" values="InProgress" operand="Status" operator="Equals" values="Queued" `
    --last-updated-after ((Get-Date).AddDays(-1) | Get-Date -Format "o" -AsUTC) `
    --last-updated-before (Get-Date -Format "o" -AsUTC)

az datafactory pipeline-run query-by-factory `
    --factory-name $DataFactoryName `
    --resource-group $ResourceGroupName `
    --filters operand="Status" operator="Equals" values="InProgress" values="Queued" `
    --last-updated-after ((Get-Date).AddDays(-1) | Get-Date -Format "o" -AsUTC) `
    --last-updated-before (Get-Date -Format "o" -AsUTC)

az datafactory pipeline-run query-by-factory `
    --factory-name $DataFactoryName `
    --resource-group $ResourceGroupName `
    --filters operand="Status" operator="Equals" values="InProgress,Queued" `
    --last-updated-after ((Get-Date).AddDays(-1) | Get-Date -Format "o" -AsUTC) `
    --last-updated-before (Get-Date -Format "o" -AsUTC)

az datafactory pipeline-run query-by-factory `
    --factory-name $DataFactoryName `
    --resource-group $ResourceGroupName `
    --filters operand="Status" operator="Equals" values="InProgress Queued" `
    --last-updated-after ((Get-Date).AddDays(-1) | Get-Date -Format "o" -AsUTC) `
    --last-updated-before (Get-Date -Format "o" -AsUTC)

Issue script & Debug output

N/A

Expected behavior

I expect the features described in the documentation to be available and readable.

Environment Summary

azure-cli 2.53.0 *

core 2.53.0 * telemetry 1.1.0

Extensions: datafactory 0.9.0

Dependencies: msal 1.24.0b2 azure-mgmt-resource 23.1.0b2

Python (Windows) 3.10.10 (tags/v3.10.10:aad5f6a, Feb 7 2023, 17:05:00) [MSC v.1929 32 bit (Intel)]

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[hwwilliams]

Any update on this?

[hwwilliams]

I have also tried using the operator Contains as I saw in other parts of the Azure CLI that use the --filters option that this operator works but it failed to work here as I get an error.

I saw on this line that the operator In may be allowed which does not result in an error but then I don't know how to write the values="" argument to accept multiple values.

[EntityAdam]

I'm no python expert, but it appears as there isn't a way for argparse to allow multiple values in this manner (CSV example):

--filters operand="Status" operator="Equals" values="InProgress,Queued"

Since there's no way to pass in multiple values here, this does not allow anyone to use the 'In' operator.

It seems other parameters in the extension are allowing json objects as values which could resolve this issue.

ex;

PowerShell

--filters "@filter1.json" # az cli @<file> syntax

filter1.json

{
  "values": [
    "IngestPipeline",
    "TransformPipeline"
  ],
  "operand": "PipelineName",
  "operator": "In"
}

[EntityAdam]

If you're looking for a work-around, dropping down to the AzureRM REST API is always an option:

AzureRM REST with PowerShell example

# requires: Az modules
#  -- Install-Module -Name Az -Repository PSGallery -Force
# requires: Log in
#  -- Connect-AzAccount

$subscriptionId = "{id}"
$resourceGroupName = "{rgp-name}"
$factoryName = "{df-name}"
$lastUpdatedAfter = ((Get-Date).AddDays(-1) | Get-Date -Format "o" -AsUTC)
$lastUpdatedBefore = (Get-Date -Format "o" -AsUTC)
$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/queryPipelineRuns?api-version=2018-06-01"

$filters = @(
    @{
      operand = "Status"
      operator = "equals"
      values = @("Failed")
    }
    @{
      operand = "PipelineName"
      operator = "In"
      values = @("MyIngestPipeline", "MyTransformPipeline")
    }
)

$body = @{
  lastUpdatedAfter = $lastUpdatedAfter
  lastUpdatedBefore = $lastUpdatedBefore
  filters = $filters
} | ConvertTo-Json -Depth 3


$response = Invoke-AzRestMethod -Path ${path} -Method POST -Payload $body
$response.content