Pode Secrets Issue

[port-43]

Describe the Bug

I have a REST api that has a route that generates a signed jwt. The secret is referenced using the $secret:variableName format. The secret is stored in a secret store using Microsoft.PowerShell.SecretStore with -UnlockInterval of 1. This secret is also mounted using Mount-PodeSecret with a -CacheTtl of 5.

This route is called on a schedule by a client to retrieve a fresh jwt. Occasionally when calling this route it receives a 500 server error with the following details (output from a json logger):

{"Date":"2024-03-06T13:45:01.0251846+00:00","Level":"Error","Server":"id-generator-8546fc9498-sbxtp","ThreadId":1,"Category":"InvalidOperation: (Microsoft.PowerShel…xtensionVaultModule:ExtensionVaultModule) [Get-Secret], PSInvalidOperationException","Message":"Unable to get secret jwtSigningKey from vault SecretStore","StackTrace":"at Get-PodeSecretManagementKey, /usr/local/share/powershell/Modules/Pode/Private/Secrets.ps1: line 203\nat Get-PodeSecret, /usr/local/share/powershell/Modules/Pode/Public/Secrets.ps1: line 565\nat <ScriptBlock>, <No file>: line 15\nat Invoke-PodeScriptBlock, /usr/local/share/powershell/Modules/Pode/Public/Utilities.ps1: line 530\nat <ScriptBlock>, <No file>: line 101"} {"Date":"2024-03-06T13:45:01.0283703+00:00","Level":"Error","Server":"id-generator-8546fc9498-sbxtp","ThreadId":1,"Category":null,"Message":"Unable to get secret from vault SecretStore","StackTrace":null} {"Date":"2024-03-06T13:45:01.0356015+00:00","Level":"Error","Server":"id-generator-8546fc9498-sbxtp","ThreadId":1,"Category":"System.Management.Automation","Message":"The pipeline was not run because a pipeline is already running. Pipelines cannot be run concurrently.","StackTrace":" at System.Management.Automation.Runspaces.PipelineBase.DoConcurrentCheck(Boolean syncCall, Object syncObject, Boolean isInLock)\n at System.Management.Automation.Runspaces.RunspaceBase.DoConcurrentCheckAndAddToRunningPipelines(PipelineBase pipeline, Boolean syncCall)\n at System.Management.Automation.Runspaces.PipelineBase.CoreInvoke(IEnumerable input, Boolean syncCall)\n at System.Management.Automation.Runspaces.PipelineBase.Invoke(IEnumerable input)\n at System.Management.Automation.PowerShell.Worker.ConstructPipelineAndDoWork(Runspace rs, Boolean performSyncInvoke)\n at System.Management.Automation.PowerShell.Worker.CreateRunspaceIfNeededAndDoWork(Runspace rsToUse, Boolean isSync)\n at System.Management.Automa

Steps To Reproduce

Steps to reproduce the behavior:

1. Setup a pode server with a secret store using Microsoft.PowerShell.SecretStore and an unlock interval of 1
2. Mount a secret for jwt signing with a catch ttl of 5
3. Reference the secret in a route using $secret:variableName
4. Call route on a schedule
5. See error

Expected Behavior

Pode retrieves the secret from internal cache and/or from the vault directly.

Screenshots

If applicable, add screenshots to help explain your problem.

Platform

• OS: Running in Kubernetes
• Browser: n/a
• Versions:
  • Pode: 2.9.0-alpine
  • PowerShell: PS 7.4.1

Additional Context

Add any other context about the problem here.

[mdaneri]

Can you post a sample code to reproduce the issue?

[port-43]

My apologies for the delay. Here's some example code for how the api is setup and loading/referencing secrets:

server.ps1

{
    Add-PodeEndpoint -Address * -Port 8080 -Protocol Http

    # set pode view engine
    Set-PodeViewEngine -Type Pode

    # setup logging
    New-PodeLoggingMethod -Terminal -Batch 10 -BatchTimeout 5 | Enable-PodeRequestLogging -UsernameProperty 'name'

    # custom json structured error logging method
    $JsonErrorMethod = New-PodeLoggingMethod -Custom -ScriptBlock {
        param($item)
        [ordered]@{Date = $item.Date; Level = $item.Level; Server = $item.Server; ThreadId = $item.ThreadId; Category = $item.Category; Message = $item.Message; StackTrace = $item.StackTrace} | Convertto-json -compress | Out-default
    }

    $JsonErrorMethod | Enable-PodeErrorLogging -Raw

    # register secret vault
    Register-PodeSecretVault -Name 'SecretStore' -ModuleName Microsoft.PowerShell.SecretStore -UnlockSecret $(Get-Content $env:VAULT_KEY_PATH) -UnlockInterval 1 -VaultParameters @{
        Authentication  = "Password"
        Interaction     = "None"
        Password        = $(Get-Content $env:VAULT_KEY_PATH | ConvertTo-SecureString -AsPlainText -Force)
        PasswordTimeout = 80
        Scope           = "CurrentUser"
    }

    # initialize secrets
    Use-PodeScript -Path './scripts/initialize-secrets.ps1'

    # mount secrets
    Mount-PodeSecret -Name "jwtSigningKey" -Vault 'SecretStore' -Key 'jwtSigningKey' -CacheTtl 5

    # initialize pode state
    Use-PodeScript -Path './scripts/initialize-state.ps1'

    # source access methods
    Use-PodeAccess -Path './access'

    # source auth methods
    Use-PodeAuth -Path './auth'

    # source routes
    Use-PodeRoutes -Path './routes' -IfExists Skip
}

initialize-secrets.ps1

Set-Secret -Name jwtSigningKey -Secret $Secret

login route initialized by Add-PodeRoute -Path 'path/to/file.ps1

{
    $Header = @{
        alg = 'hs256'
        typ = 'JWT'
    }

    $Payload = @{
        iss   = 'http://127.0.0.1/'
        sub   = $WebEvent.Auth.User.id
        name  = $WebEvent.Auth.User.name
        roles = $WebEvent.Auth.User.roles
        exp   = ([System.DateTimeOffset]::Now.AddMinutes(10).ToUnixTimeSeconds())
    }
    Write-PodeJsonResponse -Value @{
        jwt = ConvertTo-PodeJwt -Header $Header -Payload $Payload -Secret $secret:jwtSigningKey
    }
}

[Badgerati]

This was being caused by a threading issue, when either a Get/Set or Get/Unlock were being called simultaneously then the following error was thrown, which then caused the Unable to get secret error to be thrown.

The pipeline was not run because a pipeline is already running. Pipelines cannot be run concurrently

I've added in locking around Get, Set, Unlock, Read, Update, and Remove to make secrets thread safe.