Framework handles skipped activities differently than Fabric environment

[RowinVeneman]

The framework handles skipped activities differently than the actual Fabric environment. In Fabric, attempting to access the status of a skipped activity (activity('Notebook').status) causes the expression to break. However, the testing framework returns None. This means that an expression like:

@contains(
  createArray(
    activity('Fail1').status,
    activity('Notebook').status
  ),
  'Failed'
)

in the if-condition of this pipeline:

will work differently in tests compared to production. My preferred solution would be for Fabric to have a 'Skipped' status for skipped activities, but that is a different matter. For the testing framework to match Fabric it should fail to evaluate the if condition.

For completeness, this is the error:

[LeonardHd]

Hi @RowinVeneman,

Thanks for your report. Let me have a look into the expression evaluation and try to reproduce the issue. We will keep you posted.

Thank you!

[SemUijen]

Hi @LeonardHd

Do you have an update on this issue?

[LeonardHd]

Hi @SemUijen,

Thank you for bumping this up and for your patience! I can reproduce your scenario and see indeed that we can improve/fix things:

1. Functional tests do not allow you to cover your test as we stop evaluating the pipeline after a Fail activity. This is something we could improve - thanks for making us aware of this. FYI: @arjendev

2. A bug in the evaluation. We currently always pass a status property to the expression evaluation. A fix should be rather straightforward and I will track this with #164 - I will try to prioritize this issue.

I would expect that you want to test such a scenario:

def test_should_raise_expression_error(request: pytest.FixtureRequest) -> None:
    # Arrange
    fabric_folder = Path(request.fspath.dirname, "fabric")
    test_framework = TestFramework(framework_type=TestFrameworkType.Fabric, root_folder_path=fabric_folder)
    pipeline = test_framework.get_pipeline_by_name("ExamplePipeline")
    activity = pipeline.get_activity_by_name("If Condition")
    state = PipelineRunState(
        activity_results=[
            ActivityResult(
                activity_name="Fail",
                status=DependencyCondition.FAILED,
                output={
                    "outputName": "value",
                },
            ),
            ActivityResult(
                activity_name="Notebook",
                status=None, # if none do not add property
                output={},
            ),
        ]
    )

    # Act
    activity.evaluate(state)

    # Assert
    
    # We should not reach this point as the expression should raise an error
    assert False, "Expected ControlActivityExpressionEvaluatedNotToExpectedTypeError to be raised"

[SemUijen]

Hi @LeonardHd

Thx for the update!

1. What do you mean with "Functional tests do not allow you to cover your test"?
2. This fix will definitely solve the difference in evaluation between Fabric and the test framework, great!

We want to change our expression to this:

@contains(
  createArray(
    activity('Fail1').status,
    activity('Notebook')?.status
  ),
  'Failed'
)

To still test if an activity is skipped and thus is empty(i.e. {})

[LeonardHd]

Hi @SemUijen,

Regarding (1) : We have two types of tests. You can write activity or pipeline tests (we called them functional tests before - sorry for the confusion). The fix in #165 would only allow to test your scenario using an activity test. @arjendev and I are currently in discussion whether we want to proceed with the fix in #165 as it changes the concept of ActivityResult.

Regardless of whether we accept #165 or come up with another way of supporting this type of test we see the gap and try to address it as soon as possible. Thank you for your patience!