feat(python): add Python worker support - POC

[beydogan]

Related #862 and full python support from roadmap

This is a POC and discussion PR for full Python worker support. I leveraged Claude Code (supervised, reviewed and tested) to create a quick POC to validate the approach before investing significant time, since I wasn't sure if you'd be okay with this direction. Sorry about the PR size, but this was the minimum version needed to make it functional.

I used gRPC for Node↔Python IPC, which opens the door for more runtimes like Go, Ruby, etc.

I'm aware it needs more polishing, cleanup, testing, and additional features to be production-ready. I'm open to discussion and feedback. Please let me know what you think!

Feel free to close if it doesn't fit your vision or approach, but I hope we can make it work because I believe full Python support would benefit the community. I've found current python support very limited and hard to work with in complex workflows.

What works? (only dev mode)

• Task indexing and registering
• Running simple python only tasks with payloads and configs like max_duration etc.

What needs to be done?

• Containerfile generation for managed and self-hosted support
• Worker log streaming
• SDK support for triggering
• Polishing and cleaning up python runtime related code
• Probably more stuff.

Some flows to visualize the differences: Build Time: Index Phase Execution Phase

✅ Checklist

• [x] I have followed every step in the contributing guide
• [x] The PR title follows the convention.
• [x] I ran and tested the code works

---

Testing

Tested with references/hello-world-python:

• Task registration and indexing
• Task execution via gRPC
• Structured logging
• Error handling
• Build and dev commands

---

Changelog

Added Python worker support with gRPC communication.

• Implemented Python SDK with task decorator, IPC layer, execution context
• Created index and run workers for Python task discovery and execution
• Added Node.js stdio IPC bridge for spawning Python processes
• Integrated Python runtime into task execution flow
• Added Python build system with file discovery and requirements.txt handling
• Migrated workers from stdio to gRPC for better reliability
• Fixed max_duration conversion (seconds → ms) and PYTHONPATH handling
• Disabled bytecode cache in dev mode for fresh reloads

---

Screenshots

Indexed tasks:

Samples from running the hello-task-v2

[changeset-bot[bot]]

⚠️ No Changeset found

Latest commit: b25521c5baab440d2e95c7df302491065d4f6548

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[coderabbitai[bot]]

Walkthrough

This pull request introduces comprehensive Python runtime support to the Trigger.dev platform. It adds a new Python SDK package with task registration, IPC communication, and schema definitions; integrates Python worker execution into the CLI v3 build system, deployment infrastructure, and dev supervisor; implements gRPC and stdio-based IPC protocols for worker-coordinator communication via a new protobuf schema; creates Python entry point workers for task indexing and execution; and includes build tooling for Python bundling and requirements.txt parsing. The changes span configuration files, TypeScript/JavaScript implementations for the CLI, a complete Python SDK package, Python worker scripts, and integration tests.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

Areas requiring extra attention:

• packages/core/proto/worker.proto — New protobuf schema defining worker-coordinator message contracts; requires verification that all message types align with implementations on both TypeScript and Python sides
• packages/cli-v3/src/ipc/grpcServer.ts — Complex gRPC server with state management, connection tracking, and bidirectional streaming; verify error handling, cleanup on shutdown, and connection lifecycle semantics
• packages/python-sdk/trigger_sdk/ipc/grpc.py — gRPC client implementation with async/await patterns, proto-to-pydantic conversions, and connection retry logic; ensure message conversion correctness and lifecycle alignment with server
• packages/cli-v3/src/build/pythonBundler.ts — Python bundling logic including manifest generation, file copying, and requirements.txt handling; verify output directory structure and manifest schema compliance
• packages/cli-v3/src/python/pythonTaskRunner.ts — Task execution lifecycle with timeout handling, signal management, and result conversion; verify proper cleanup and error propagation
• packages/cli-v3/src/entryPoints/python/managed-run-worker.py — Python worker entry point with gRPC communication, signal handling, and context management; check for potential deadlocks or resource leaks
• Proto conversion layers (protoCompletionToResult, executionToProtoMessage, _proto_to_pydantic, _pydantic_to_proto) — Bidirectional message transformations between protobuf and native types; verify all field mappings are bidirectional and handle optional fields correctly
• IPC integration points — Multiple files integrating IPC into existing flows (dev/devSupervisor.ts, dev/backgroundWorker.ts, entryPoints/dev-run-controller.ts, entryPoints/managed/execution.ts, indexing/indexWorkerManifest.ts); verify conditional logic for Python runtime doesn't break existing Node/Bun paths
• Python dependencies parsing (trigger_sdk/schemas/resources.py, pythonDependencies.ts) — Requirements.txt parsing and validation logic; verify regex patterns handle edge cases (extras, version ranges, comments)
• Task registry and execution (trigger_sdk/task.py, pythonTaskRunner.ts) — Global state management and task lifecycle; verify no race conditions or state leaks across concurrent executions
• Test fixtures and integration tests — Verify test setup correctly mocks gRPC connections, handles timeouts, and validates end-to-end flows without flakiness

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 74.86% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The PR title 'feat(python): add Python worker support - POC' clearly and specifically describes the main change: adding Python worker support as a proof-of-concept, using the conventional feat prefix and appropriate scope.
Description check	✅ Passed	The PR description includes the checklist items (all checked), provides context (related issues/roadmap), explains what works (dev mode, indexing, simple task execution), what's needed (containerfile generation, logging, SDK features), includes testing details, provides a detailed changelog, and includes screenshots demonstrating the feature. The description is comprehensive and substantially complete.

✨ Finishing touches

• [ ] 📝 Generate docstrings

🧪 Generate unit tests (beta)

• [ ] Create PR with unit tests
• [ ] Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[ericallam]

@beydogan this is very interesting! Thanks for the proposal and showing a working proof of concept! I think the very first thing we would need to do before introducing a full python runtime would be to produce a full python SDK (similar to our @trigger.dev/sdk). Additionally, when introducing something like this the consideration isn't only the code being committed, but then we would have to keep the python runtime up to date whenever working on something new. This would be too big of a burden at the moment. For that reason I'm going to close this PR for now, but hopefully one day we'll be able to resurrect it and provide a full python runtime like proposed here.