RaiseEvent-Call doesn't transmit content/body to Orchestrator

[amazingidiot]

Hello, I'm running a durable function with the human interaction pattern like in the human interaction example . I successfully raise the event in a separat azure function via a HTTP call according to the documentation. In line 22, $approvalEvent, the return value of the Start-DurableExternalEventListener is used as the input for the activity. I assume that that's the value of the body that's send with the HTTP-call to raise the event, although it is not actually used in the activity itself in the example. I managed to capture the message in the queue, and the body is in there under TaskMessage.Event.Input. If I convert the $approvalEvent to JSON, it is just "{}". Is the message body not available in powershell or is there a cmdlet to query the data? The documentation is incomplete on this so far.

Please let me know if I can provide further information! Thank you!

Best regards, Oliver

[amazingidiot]

Hi, any news on this issue? Do you need more information from me? Thanks!

[davidmrdavid]

Hi @amazingidiot,

No updates yet, sorry to keep you waiting. I've assigned this ticket to me to help track it and I'll start looking into it as soon as possible. It seems to me, from looking at the description, that this is a bug in the SDK implementation, but I'll need a few days of exploration before being able to make that determination. We'll update this thread as soon as possible.

[amazingidiot]

Hi @davidmrdavid, thank you for the reply! Again, if I can provide more information, please tell me.

Best regards!

[oobegreg]

@amazingidiot I was also having the same issue as you so I decided to look in to the issue.

When Wait-DurableTask is used with the -Any parameter a task is returned to PowerShell not the resulting job status, output or event. This is a different behaviour to Wait-DurableTask without the -Any parameter. In this case it returns an array of all the results. This is why you and I were both seeing {} returned.

To ensure compatibility with existing code and logic I created a new command Get-DurableTaskResult. To this you pass a task to get the results. You can pass any task you like (that has completed), but it's best to pass the output of Wait-DurableTask as this will return the first completed task (as per the logic for the -Any paramater).

Below you can find the modified demo HelloOrchestration logic. You can see it contains a timeout task, the main activity task and two event tasks. I then wait for any of these to trigger in a loop. If the activity finishes the loop completes and the Orchestration is completed.

using namespace System.Net

param($Context)

Set-DurableCustomStatus -CustomStatus 'Processing Tokyo'

Write-Host "Start Activity"
$ParallelActivity = Invoke-DurableActivity -FunctionName 'Hello' -Input 'Tokyo' -NoWait
Write-Host "Activity started"

$ActivityComplete = $false
while (!$ActivityComplete) {
    # Set a timeout
    $ApprovalTimeOut = New-TimeSpan -Seconds 60
    $DurableTimeoutEvent = Start-DurableTimer -Duration $ApprovalTimeOut -NoWait

    # Set events to listen for
    $DurableActivityUpdate = Start-DurableExternalEventListener -EventName "TESTEVENTNAME" -NoWait
    $DurableActivityUpdate2 = Start-DurableExternalEventListener -EventName "TESTEVENTNAME2" -NoWait

    # Wait for event or task to complete
    $CompleteRunningTask = Wait-DurableTask -Task @($DurableActivityUpdate, $DurableActivityUpdate2, $ParallelActivity, $DurableTimeoutEvent) -Any

    # Event Raised
    if ($CompleteRunningTask -eq $DurableTimeoutEvent) {
        Write-Host "Timeout!"
        $ActivityComplete = $true
        Stop-DurableTimerTask -Task $DurableTimeoutEvent
        return
    }
    if ($CompleteRunningTask -eq $DurableActivityUpdate) {
        Write-Host "status update"
        Write-Host $(Get-DurableTaskResult -Task @($CompleteRunningTask) | ConvertTo-Json -Depth 100)
    }
    if ($CompleteRunningTask -eq $DurableActivityUpdate2) {
        Write-Host "status update 2"
        Write-Host $(Get-DurableTaskResult -Task @($CompleteRunningTask) | ConvertTo-Json -Depth 100)
    }
    if ($CompleteRunningTask -eq $ParallelActivity) {
        Write-Host "Activity Complete"
        Write-Host $(Get-DurableTaskResult -Task @($CompleteRunningTask) | ConvertTo-Json -Depth 100)
        $ActivityComplete = $true
        Stop-DurableTimerTask -Task $DurableTimeoutEvent
    }
}

Write-Host "Orchestration complete"

$output

Below is the Hello activity. You will see I've created a custom Send-DurableExternalEvent function as the inbuilt function doesn't work on Azure due to the App code not being specified. I have added that as a parameter on this commit too.

using namespace System.Net

param($Request, $TriggerMetadata)

"Hello $Request!"

# Set instance ID
$InstanceId = $TriggerMetadata.InstanceId

# Set durable function information for custom Send-DurableExternalEvent
$FunctionsURL = "http://localhost:7071"
$AppCode = "XXX"

function Send-DurableExternalEvent() {
    param (
		[Parameter(Mandatory = $true)]
		[string]$FunctionsURL,

		[Parameter(Mandatory = $true)]
		[string]$InstanceId,
		
		[Parameter(Mandatory = $true)]
		[string]$AppCode,
		
		[Parameter(Mandatory = $true)]
		[string]$EventName,

		[Parameter(Mandatory = $false)]
		[object]$EventDataJson
	)

    $EventURL = ("{0}/runtime/webhooks/durabletask/instances/{1}/raiseEvent/{2}?code={3}" -f $FunctionsURL, $InstanceId, $EventName, $AppCode)

    try {
        $null = Invoke-WebRequest -Method POST -Uri $EventURL -body $EventDataJson -ContentType 'application/json' -Headers @{"accept" = "application/json" }
        Write-Host "Send-DurableExternalEvent success"
    }
    catch {
        Write-Host "Send-DurableExternalEvent failed"
    }
    
}

Write-Host "Trigger event 1"
Send-DurableExternalEvent -FunctionsURL $FunctionsURL -InstanceId $InstanceId -AppCode $AppCode "TESTEVENTNAME" -EventDataJson "{'event': 'first event'}"
Start-Sleep 2

Write-Host "Trigger event 2"
Send-DurableExternalEvent -FunctionsURL $FunctionsURL -InstanceId $InstanceId -AppCode $AppCode "TESTEVENTNAME2" -EventDataJson "{'event': 'second event'}"
Start-Sleep 4

Write-Host "Hello finished"

@davidmrdavid I have created a fork and commit here https://github.com/oobegreg/azure-functions-powershell-worker/tree/3160e043732836c8f6af7d9e76ce7a9e9186d390 as I wasn't sure how to properly contribute to this repo.

It contains the updated Send-DurableExternalEvent function to handle the Azure Functions app code when deployed to Azure. It also contains updated logic to support event triggers and a new command Get-DurableTaskResult.

Hopefully this helps and could be deployed to Azure as soon as possible.

[amazingidiot]

Thank you a lot for the information!

[oobegreg]

@ConnorMcMahon @davidmrdavid Happy new year. Are you able to provide any timeframes on when this might (or an appropriate fix) may be pushed to Azure production? I need to understand the impacts for our production deployment capabilities.

Thank you

[davidmrdavid]

Hi @oobegreg,

Thank you for reaching out. I'll bring this up for triaging today and I should be able to get back to you with an update, apologies for our delay here.

cc/ @AnatoliB, for context.

[davidmrdavid]

Hi @oobegreg,

I want to make sure we have the same starting context: when using the -Any flag, the result of a Wait-DurableTask should return a Task object instead of the result. You pointed this out in your description, but I wanted to make sure we were in the same page on this being the intended behavior :) .

With that in mind, the missing step would then be to obtain the result data from within that Task. I haven't managed to test this yet, but I would have expected that calling Wait-DurableTask again on that object should yield the result. I do realize we should have a better means of doing this, I'm still exploring whether that already exists in the codebase.

If using Wait-DurableTask on the task object doesn't work, then I do think we should incorporate your Get-DurableTaskResult command, or something very much like it.

@oobegreg: had you tried using Wait-DurableTask on the result of Wait-DurableTask with -Any? Did that work for you?

As for a timeline for a fix, I think the earliest I could merge a fix would be around Wednesday next week. After that, we're tied to the release schedule of the PowerShell Azure Functions worker, and that can take a few weeks. That's why I want to first explore whether there are any means of extracting this result from within the existing codebase as well; it can be a fair wait otherwise.

[oobegreg]

Hi @davidmrdavid,

Thanks for the quick response and assistance. Not only is it great, it's very much appreciated too.

I think using the -Any flag with Wait-DurableTask should return a task. That way it's easy to determine which task responded by the statement if ($CompleteRunningTask -eq $DurableTimeoutEvent) {.

Thanks for trying to find alternative options in the existing code too. I tried Wait-DurableTask again after the initial call with Any and it crashes (I assume it crashes as the code doesn't continue, but there's no error).

I modified the example code above to look like the following. In the console output I see OUTPUT 1 returned on the log, but nothing after that.

if ($CompleteRunningTask -eq $ParallelActivity) {
        Write-Host "Activity Complete"
        Write-Host "OUTPUT 1"
        $ActivityResults = Wait-DurableTask -Task @($ParallelActivity)

        Write-Host "OUTPUT 2"
        Write-Host ($ActivityResults.GetType())
        Write-Host ($ActivityResults | ConvertTo-Json -Depth 100)
        Write-Host "OUTPUT 3"

        # Write-Host $(Get-DurableTaskResult -Task @($CompleteRunningTask) | ConvertTo-Json -Depth 100)
        $ActivityComplete = $true
        Stop-DurableTimerTask -Task $DurableTimeoutEvent
    }

I also tried Wait-DurableTask -Task $ParallelActivity for good measure.

I think it's not working because the task is already marked as processed/played and no longer returned when Wait-DurableTask is called again (and by extension GetCompletedHistoryEvent).

internal override HistoryEvent GetCompletedHistoryEvent(OrchestrationContext context, HistoryEvent taskScheduled, bool processed)
        {
            return context.History.FirstOrDefault(
                    e => e.EventType == HistoryEventType.EventRaised &&
                         e.Name == ExternalEventName &&
                         e.IsPlayed == processed);
        }

Just an FYI, I tried this with both the modified and production 4.0.1.16815 codeset to be sure.

Is it's possible to include an update to Send-DurableExternalEvent as well (also part of my commit) to ensure that function works in Azure? If so, that would be amazing.

Those suggested timeframes are great too.

Thanks again, Greg

[oobegreg]

Hi @davidmrdavid have you been able to progress this? Thanks

[davidmrdavid]

Hi @oobegreg, so sorry for my delayed response, there's been a lot going on, so I actually do have quite a bit to share.

In short, the original issue described in this thread (external events not returning their input when the task completes) can be fixed independently on the Get-DurableTaskResult CmdLet you proposed. In fact, I have a working branch where I've included the fix here: https://github.com/Azure/azure-functions-powershell-worker/blob/506e4f78564fe0b3d71bf1e7705c9bf522c17ce5/src/DurableSDK/DurableTaskHandler.cs#L64-L71

However, your proposed CmdLet (and associated changes in your fork) is in fact a missing feature that we urgently need, and I do want to merge it into the repo (I really appreciate your contribution and the implementation seems sound!), but given that it represents a new public API, and the fact that PowerShell's Durable Functions APIs is already so unique, it will require just a bit more time before we're ready to do so. Given that I don't have direct control over these timelines, I'd like to avoid giving an ETA on that to properly manage expectations here. That said, I've opened a PR to merge your changes here - https://github.com/Azure/azure-functions-powershell-worker/pull/753

I also want to clarify where I've been spending my development time here, because I realize that this particular issue may seem like an easy fix . In general, as I mentioned earlier in this thread, the team is investing heavily in the implementation of a new deployment strategy that should allow us to release bug fixes and new features to the PowerShell DF SDK at a faster rate than is possible today, and most of my time has been invested in there. While still very much in development and not ready for review, you can see a tiny part of that effort in this PR: https://github.com/Azure/azure-functions-powershell-worker/pull/746/files

In general, we're working to extract the Durable SDK out of the worker and make it into its own package to be eventually released in the PowerShell gallery (or something like it). So in addition to the worker-level changes that exist in this PR, we have a private repo where a mirror'ed SDK is being developed. In any case, I'll be incorporating the bug reports listed in this repo as fixes in both this and the mirror'ed SDK, but right now we're prioritizing being able to have that separate and fast-to-release SDK as the first step to fixing some of these bugs. The good news though is that this week we made serious progress towards finishing that effort, so I believe it won't be long until we're in a good position to make hotfix releases without any external delays.

And that's been it! I appreciate your patience, and I'll push to get that PR of your fork merged asap

[oobegreg]

Thank you @davidmrdavid. Everything sounds very exciting for the future of the platform. I appreciate everything you and the team have been doing.

[ruankrTs]

I am seeing some odd behaviour (also raised as a new issue here )

HttpTrigger

using namespace System.Net

param($Request, $TriggerMetadata)

$inputs = @{
    Name = "$($Request.Query.Name)"
}

$FunctionName = $Request.Params.FunctionName
$InstanceId = Start-DurableOrchestration -FunctionName $FunctionName -Input $inputs 
Write-Host "Started orchestration with ID = '$InstanceId'"

$Response = New-DurableOrchestrationCheckStatusResponse -Request $Request -InstanceId $InstanceId
Push-OutputBinding -Name Response -Value $Response

Orchestrator

using namespace System.Net
param($Context)

$output = @()
$gate1 = Start-DurableExternalEventListener -EventName "Paris" -NoWait -verbose
$gate2 = Start-DurableExternalEventListener -EventName "London" -NoWait -verbose

$output = Invoke-DurableActivity -FunctionName 'Hello' -Input $Context.Input

$endResults = Wait-DurableTask -Task @($gate1, $gate2)

$finaloutput += Invoke-ActivityFunction -FunctionName 'Bye' -Input $output

$finaloutput

Hello (Activity)

using namespace System.Net
param($name, $TriggerMetadata)

$InstanceId = $TriggerMetadata.InstanceId

Write-Host "Hello $name"

Send-DurableExternalEvent -InstanceId $InstanceId -EventName "London" -verbose
Send-DurableExternalEvent -InstanceId $InstanceId -EventName "Paris" -verbose

$name

Bye (Activity)

using namespace System.Net
param($name)

Write-Host "Bye $name"

$name

It seems there is a breaking change in the latest version of PowerShell for the Wait-DurableTask functionality. Using the above code when I do:

"FUNCTIONS_WORKER_RUNTIME_VERSION": "~7"

It works and will run Bye and produce the correct output. However, when I change it to:

"FUNCTIONS_WORKER_RUNTIME_VERSION": "7.2"

It stays in the running state and does not run Bye. Sending the 2 events manually again, it throws an exception below:

 Orchestration completed with a 'Failed' status and 314 bytes of output. Details: Unhandled exception while executing orchestration: DurableTask.Core.Exceptions.NonDeterministicOrchestrationException: Non-Deterministic workflow detected: A previous execution of this orchestration scheduled an activity task with sequence ID 1 and name 'Bye' (version ''), but the current replay execution hasn't (yet?) scheduled this task. Was a change made to the orchestrator code after this instance had already started running?
[2022-08-26T06:44:07.415Z]    at DurableTask.Core.TaskOrchestrationContext.HandleTaskScheduledEvent(TaskScheduledEvent scheduledEvent) in /_/src/DurableTask.Core/TaskOrchestrationContext.cs:line 271
[2022-08-26T06:44:07.415Z]    at DurableTask.Core.TaskOrchestrationExecutor.ProcessEvent(HistoryEvent historyEvent) in /_/src/DurableTask.Core/TaskOrchestrationExecutor.cs:line 189
[2022-08-26T06:44:07.416Z]    at DurableTask.Core.TaskOrchestrationExecutor.<ExecuteCore>g__ProcessEvents|11_0(IEnumerable`1 events) in /_/src/DurableTask.Core/TaskOrchestrationExecutor.cs:line 114
[2022-08-26T06:44:07.416Z]    at DurableTask.Core.TaskOrchestrationExecutor.ExecuteCore(IEnumerable`1 pastEvents, IEnumerable`1 newEvents) in /_/src/DurableTask.Core/TaskOrchestrationExecutor.cs:line 122
[2022-08-26T06:44:07.417Z]         at DurableTask.Core.TaskOrchestrationContext.HandleTaskScheduledEvent(TaskScheduledEvent scheduledEvent) in /_/src/DurableTask.Core/TaskOrchestrationContext.cs:line 271
[2022-08-26T06:44:07.417Z]    at DurableTask.Core.TaskOrchestrationExecutor.ProcessEvent(HistoryEvent historyEvent) in /_/src/DurableTask.Core/TaskOrchestrationExecutor.cs:line 189
[2022-08-26T06:44:07.420Z]    at DurableTask.Core.TaskOrchestrationExecutor.<ExecuteCore>g__ProcessEvents|11_0(IEnumerable`1 events) in /_/src/DurableTask.Core/TaskOrchestrationExecutor.cs:line 114
[2022-08-26T06:44:07.420Z]    at DurableTask.Core.TaskOrchestrationExecutor.ExecuteCore(IEnumerable`1 pastEvents, IEnumerable`1 newEvents) in /_/src/DurableTask.Core/TaskOrchestrationExecutor.cs:line 122

If I make the orchestrator slightly complicated (chain functions) then it behaves even more inconsistently. Anyone else have success running external events with PowerShell 7.2?

[misteriks]

Today I ran into the exact same issue that @amazingidiot reported in 2021. Missing the http body when using -NoWait on Start-DurableExternalEventListener and Start-DurableTimer and -any on Wait-DurableTask

There has been some activity on this issue, but it is not entirely clear to me if this has been resolved or not.

I'm running Core Tools 4.0.5312, Function Runtime 4.23.2.21220 and PSVersion 7.3.7