ADF pipeline config_ExportData failing: property doesn't exist error

[digitalsb]

🐛 Problem

Getting this error while running the ADF pipeline config_ExportData

Error code ActionFailed Failure type User configuration issue Details Activity failed because an inner activity failed; Inner activity name: Run exports for scope, Error: Operation on target ForEach export scope failed: Activity failed because an inner activity failed; Inner activity name: If scheduled, Error: The expression 'and(equals(toLower(item().properties.schedule.recurrence), toLower(pipeline().parameters.Recurrence)),startswith(toLower(item().name), toLower(variables('hubName'))))' cannot be evaluated because property 'recurrence' doesn't exist, available properties are 'status'. Activity ID faa571d8-a5dc-41d2-90c8-a532bd10ea3f Error source Pipeline config_ExportData

📷 Screenshots

ℹ️ Additional context

TODO: Add any other context about the problem here. Remove if not applicable.

🙋‍♀️ Ask for the community

We could use your help:

1. Please vote this issue up (👍) to prioritize it.
2. Leave comments to help us solidify the vision.

[arthurclares]

@flanakin Is this really for September or was that a mistake?

[flanakin]

@arthurclares I added it to September to make sure we look into it.

[flanakin]

@santoshblearner Looking at this, I don't see an issue with the default configuration. Do you think the pipeline might have been updated accidentally? Can you confirm the config_RunExports pipeline has the Recurrence parameter?

1. Open the hub resource group
2. Select the Data Factory instance
3. Select the Launch studio button
4. Select the Author (pencil) icon on the left
5. Select the config_RunExports pipeline
6. Confirm the parameters in the Parameters tab:

[digitalsb]

hi @flanakin Thanks for your comments. I debugged this issue over the weekend and found the fix.

Root Cause The issue arises in the config_RunExports pipeline during the evaluation of an expression in the IfCondition activity. The pipeline tries to evaluate the following expression:

and(equals(toLower(item().properties.schedule.recurrence), toLower(pipeline().parameters.Recurrence)),startswith(toLower(item().name), toLower(variables('hubName'))))

The error occurs because not all export items returned by the Get exports for scope activity have a recurrence property in the schedule object. I had a few one-time exports configured earlier and they don't have the recurrence property. Since the function expects both conditions to be evaluated, the pipeline fails when the recurrence property is missing for one-time or unscheduled exports.

Proposed Fix The pipeline processes a set of exports where only a subset is linked to the finOpsHub. However, the pipeline is currently attempting to evaluate all exports, even those not linked to the hub, and do not have a recurrence property. As a result, the pipeline fails when encountering unscheduled exports that lack the recurrence field.

The solution is to update the IfCondition activity’s expression so that it first filters exports based on whether they are linked to finOpsHub by checking if their name starts with the hubName variable. Only after filtering the exports should the pipeline check for the presence of the recurrence property. This will ensure that irrelevant or unscheduled exports are skipped early, preventing the error.

Updated Expression:

and(startswith(toLower(item().name), toLower(variables('hubName'))), has(item().properties.schedule, 'recurrence'), equals(toLower(item().properties.schedule.recurrence), toLower(pipeline().parameters.Recurrence)))

Explanation: startswith(toLower(item().name), toLower(variables('hubName'))): This filters the exports to include only those whose name starts with the hubName (i.e., exports relevant to finOpsHub). has(item().properties.schedule, 'recurrence'): After filtering by hubName, this checks if the recurrence property exists in the export’s schedule object. equals(toLower(item().properties.schedule.recurrence), toLower(pipeline().parameters.Recurrence)): Finally, this compares the recurrence value of the export with the Recurrence parameter in the pipeline, but only if the recurrence property is present.

This change ensures that:

• Only applicable exports (those linked to finOpsHub) are evaluated further.
• The pipeline gracefully handles exports that do not have a recurrence property by skipping them.
• The expression only runs the comparison of the recurrence property when it exists, preventing the pipeline from failing on unscheduled exports.

[digitalsb]

Hi @flanakin: I am unable to assign this issue back to you. I posted the root cause and the possible fix for this issue. Please check and let me know.

[flanakin]

Thanks for the additional details! This helps a lot!

@MSBrett as FYI

[flanakin]

@MSBrett Can you look at this while you're adding managed exports support for other export types?

[MSBrett]

Will do.
FWIW this doesn't break anything, but supressing the error will be great. Thanks for the updated expression :)

[MSBrett]

Fixed in #1415