Pode Watchdog Feature (work in progress)

[mdaneri]

Summary

This pull request introduces the Pode Watchdog feature, which allows users to monitor and manage processes or scripts running within their Pode server. The Watchdog automatically tracks process status and uptime, monitors file changes, and provides API endpoints for interacting with the monitored processes. Key functionalities include process monitoring, automatic restarts, session management during restarts/shutdowns, and remote control through REST APIs.

Key Features

• Process Monitoring: Tracks the status, uptime, and performance of processes running within Pode using a NamedPipeStream.
• File Monitoring: Automatically restarts processes when monitored files (such as configuration files) are modified.
• Logging Support: Logs important events and errors for debugging and auditing purposes.
• Automatic Restarts: Ensures that monitored processes are automatically restarted if they crash or exit unexpectedly.
• Session Management During Restarts/Shutdowns: When the service is restarting or shutting down, it waits for any open sessions to terminate before proceeding. During this time, the default HTTP response code is set to 503 (Service Unavailable).
• Configurable Service Recovery: After a failure, the service will wait for a specified amount of time ($RestartServiceAfter) before attempting to restart the process. The process can be restarted a maximum of $MaxNumberOfRestarts times. The restart counter is reset after a successful run of $ResetFailCountAfter minutes.
• 503 Service Management: The Watchdog can enable or disable 503 status responses for the monitored process, providing more control over service availability during critical operations.

Usage Example

This feature allows users to set up and monitor a process with a Pode server and interact with the process via REST API routes. Below is a sample setup:

Start-PodeServer {
    # Define an HTTP endpoint
    Add-PodeEndpoint -Address localhost -Port 8082 -Protocol Http

    # Set up Watchdog logging
    New-PodeLoggingMethod -File -Name 'watchdog' -MaxDays 4 | Enable-PodeErrorLogging

    # Enable Watchdog monitoring for a script process
    Enable-PodeWatchdog -FilePath './scripts/myProcess.ps1' -FileMonitoring -FileExclude '*.log' -Name 'myProcessWatchdog'

    # Route to check process status
    Add-PodeRoute -Method Get -Path '/monitor/status' -ScriptBlock {
        Write-PodeJsonResponse -Value (Get-PodeWatchdogProcessMetric -Name 'myProcessWatchdog' -Type Status)
    }

    # Route to restart the process
    Add-PodeRoute -Method Post -Path '/cmd/restart' -ScriptBlock {
        Write-PodeJsonResponse -Value @{success = (Set-PodeWatchdogProcessState -Name 'myProcessWatchdog' -State Restart)}
    }
}

Documentation

Full documentation for the Pode Watchdog feature has been included, covering:

• Feature Overview
• Key Features
• Setup Instructions
• Process Control and Monitoring Commands
• REST API Integration

This feature needs #1387 to be complete