Prefect Recipes 🧑‍🍳 🥐

This repository contains common and extensible Prefect patterns to drive efficient workflows — we like to call these patterns our recipes

Here you'll find starter code and more advanced example use cases.

Contributing = Swag 🧢

We're always looking for new contributions! See our existing Recipe Ideas/Issues for inspiration. Read a detailed guide on how to share your solutions with the Prefect community or run these commands to get started right away.

Issues / Bugs 🐛

To report issues, typos, or link fixes, please open an issue. We appreciate it!

Recipe Ideas 👩‍🍳

What are you interested in seeing examples of? Jot down your big idea here.

Table of Contents 📖

• Getting Started

  • Introductory Videos
  • Subflows
  • Control Flow
  • Optimization
  • Notifications
  • Flow Run Observability
  • Configuration (Using Blocks
  • Parameters
  • Testing
  • Triggering Flow Runs

• Diving Deeper

  • Streaming & Event-Driven Workflows
  • Serverless
  • Data Engineering / DataOps
  • ELT/ETL
  • AWS Infrastructure
  • Azure Infrastructure
  • Helm
  • Kubernetes
  • GitHub Actions
  • Dockerfiles
  • Django
  • Pydantic
  • Hex
  • Data Science
  • Modular Data Stack
  • Miscellaneous

• Issues & Bugs

• Contributions

• Join the Discussion

• Thanks

Getting Started 🍯

Introduction

• What is Prefect?
• An Introduction to Your First Prefect Flow

Subflows

• Run Multiple Subflows or Child Flows in Parallel
• Subflow with a Different Task Runner Than Parent Flow
• Create a Subflow and Block Until It's Completed
• Running Subflows On Their Own Infrastructure Using a Separate Deployment

Control Flow

• Conditionally Stop a Task Run
• Ensure Tasks Immediately Fail If Upstream Task Fails
• Define State Dependencies Between Tasks

Optimization

• Cache a Task Result To Prevent Recomputation

Notifications

• Send Notifications with a Slack Webhook
• Sending Slack Notifications in Python with Prefect

Flow Run Observability

• Interact with REST API
• Determine Whether a Flow Run Was Executed Ad Hoc or on a Schedule

Configuration (using Blocks)

• Supercharge your Python Code with Blocks - Blog
• Supercharge your Python Code with Blocks - Video

Parameters

• Use flow parameters

Testing

• Testing Flows, Subflows, and Tasks

Logging

• Explain Your Python Exceptions with OpenAI

Triggering Flow Runs

• Triggering Flow Runs From a Deployment via API Call or From a Terminal Using Curl

Diving Deeper 🍱

Data Engineering / DataOps

• Serverless Real-Time Data Pipelines on AWS with Prefect, ECS and GitHub Actions
• Build a Data Platform with Prefect, dbt, and Snowflake (using blocks)
• Real World Python for Data Engineering - Supercharge Your Data Orchestration with Prefect 2.0
• Create a Maintainable Data Pipeline with Prefect and DVC
• Data engineering & orchestration with Prefect, Docker, Terraform, Google CloudRun, BigQuery and Streamlit

ELT/ETL

• Orchestrating Airbyte with Prefect 2
• Coordinate ELT in 2023 with Airbyte, dbt and Prefect
• Schedule & orchestrate dbt Cloud jobs with Prefect
• Prefect & Fivetran: integrate all the tools & orchestrate them in Python
• Export Airbyte Configuration and Load to S3 bucket using blocks, including Python-based deployment
• ELT with Snowflake Using Async and Blocks

AWS Infrastructure

• Deploy a Prefect agent to ECS using the AWS CLI
• Deploy a Prefect agent to ECS with Terraform
• Deploy Flows Using ECSTask Infrastructure Blocks
• Deploy a Prefect agent to ECS Fargate using CloudFormation and GitHub Actions

Azure Infrastructure

• Setup Azure with Prefect
• Deploy Prefect Orion to an AKS Cluster with Azure Blob Storage
• Setup an Azure VM and Run the Prefect Agent
• Deploy Flows Using Azure Container Instances Infrastructure Blocks

GCP Infrastructure

• Setup GCP Managed Instance Group with Prefect

Helm

• Deploy Prefect Agent using Helm and Terraform

Github Actions

• Conditionally deploy Prefect flow(s) only when flow-related files have changed
• Conditionally build a docker image & deploy Prefect flow(s) only when flow-related files have changed
• Deploy Prefect flows as containers stored in AWS ECR

Django

• Using Django with Prefect 2

Pydantic

• Using Pydantic BaseModel with Prefect 2
• Build a Full-Stack ML Application With Pydantic And Prefect

Hex

• Create Observable and Reproducible Notebooks with Hex - Article
• Create Observable and Reproducible Notebooks with Hex - Video

Data Science

• How to Structure an ML Project for Reproducibility and Maintainability
• Orchestrate Your Data Science Project with Prefect 2.0
• Build a Full-Stack ML Application With Pydantic And Prefect

Modular Data Stack

• How to Build a Modular Data Stack — Data Platform with Prefect, dbt and Snowflake
• How to Build Modular Dataflows with Tasks, Flows and Subflows in Prefect

Miscellaneous

• Merge Dependabot Pull Requests with Prefect 2 & a GitHubCredentials block

Contributions

We're always looking for new contributions! You can add your Prefect 2.0 recipe and earn some swag in a few simple steps:

1. Clone the prefect-recipes repo:

git clone [email protected]:PrefectHQ/prefect-recipes.git

2. Create and checkout a new branch:

git checkout -b feat/new-recipe-name

3. Add your code under the appropriate category. Unsure? Add it under flows-advanced/.
4. Add your recipe to this README.
5. Commit and push the code to your remote branch.
6. Create a PR 🤌

Join our discussions 🗣️

We use our Slack Community and Discourse to discuss all things Prefect-- such as FAQ, use cases and integrations. Join in the conversation :smile:

Thanks 💙

Thank you for your contributions and efforts to improve prefect-recipes. We're glad to have you in our community!