Support testing storage event triggered ADF pipelines

[thurstonchen]

As we discussed previously, it's value-added to also support testing storage event triggered ADF pipelines, therefore I create this issue to track this new feature.

And below is the rough ideas.

• For testing blob created event triggered pipelines, we'll add a new fixture to accept trigger file path, trigger name, blob event type and storage account relevant arguments to do the test with below steps:

  1. upload declared event trigger file to target container of specified storage account.
  2. get eTag and last modified time from the just uploaded event triggered file.
  3. query trigger-runs by last modified time filter and look for the one with exactly same eTag from event-payload of trigger-run.
  4. get triggered pipeline run id from the hit trigger-run, then poll the pipeline run status with the given run id, just like what the normal fixture does.

• For testing blob deleted event triggered pipelines, reuse the same new added fixture, but pass BlobDeleted to blob event type argument, then

  1. delete declared event trigger file from target storage account.
  2. since there's no eTag for file deletion, instead we query trigger-runs only by the file deletion time filter and pick the latest one with DeleteBlob data-api action from event-payload of trigger-run.
  3. get triggered pipeline run id from the hit trigger-run, then same as above, polling the pipeline run status with the given run id.

BTW, for scenarios like one ADF pipeline triggers another ones, since it could be quite different from case to case like below for instance, it's not feasible to cover all of them in pytest-adf plugin.

• pipeline A -> pipeline B
• pipeline A -> pipeline B -> pipeline C
• pipeline A -> pipeline B                  -> Pipeline C -> pipeline D
• ...

Instead, users could take use of the storage-event-trigger pipeline testing feature to do the test on separated stages one by one, and if each stage of the whole chained pipelines works as expected, then the whole one should work well too.

[A30005881]

Are these changes going to make a release? Would be extremely helpful in some of my current use-cases. @thurstonchen looking at your code, this only support local -> blob, is it possible to add (future) blob-> blob?