turbo
turbo copied to clipboard
vercel
WIP: Daemon package watch
Description
This PR allows the daemon to watch for package changes and eagerly update hashes. Each package / task maintains a list of hashes that are updated by a file watcher. When turbo launches, it queries that daemon instead of manually walking.
Still in progress:
- before we can start watching, we need to populate the state with what the existing file and package hashes are. due to the way that hashing is done right now, we do not deduplicate this across multiple tasks in the same package. this means we do M x N x G glob walks (where M is the number of tasks, N the number of packages, and G the number of input globs for that package) which can be expensive (70s on linux + turbo repo). ideally we should be able to deduplicate some of these but it still may not be enough
- switching this to evaluate lazily means that we can improve the up-front cost but lose a large chunk of time as the daemon populates the cache
- alternatively, we can do a hybrid. the daemon comes online immediately, and the initial hashing job is prioritised based on requests from the user. that way a request to hash all the build package-tasks will take priority over the other tasks
This refactor is currently in progress
Testing Instructions
This PR is mid-rebase and will get some more integration tests as I work through it.
Closes TURBO-2200
The latest updates on your projects. Learn more about Vercel for Git ↗︎
| Name | Status | Preview | Comments | Updated (UTC) |
|---|---|---|---|---|
| examples-gatsby-web | 🔄 Building (Inspect) | Visit Preview | 💬 Add feedback | Mar 1, 2024 11:46pm |
| examples-kitchensink-blog | 🔄 Building (Inspect) | Visit Preview | 💬 Add feedback | Mar 1, 2024 11:46pm |
| examples-nonmonorepo | ✅ Ready (Inspect) | Visit Preview | 💬 Add feedback | Mar 1, 2024 11:46pm |
| rust-docs | ✅ Ready (Inspect) | Visit Preview | 💬 Add feedback | Mar 1, 2024 11:46pm |
| turbo-site | ✅ Ready (Inspect) | Visit Preview | 💬 Add feedback | Mar 1, 2024 11:46pm |
6 Ignored Deployments
| Name | Status | Preview | Comments | Updated (UTC) |
|---|---|---|---|---|
| examples-basic-web | ⬜️ Ignored (Inspect) | Visit Preview | Mar 1, 2024 11:46pm | |
| examples-designsystem-docs | ⬜️ Ignored (Inspect) | Visit Preview | Mar 1, 2024 11:46pm | |
| examples-native-web | ⬜️ Ignored (Inspect) | Visit Preview | Mar 1, 2024 11:46pm | |
| examples-svelte-web | ⬜️ Ignored (Inspect) | Visit Preview | Mar 1, 2024 11:46pm | |
| examples-tailwind-web | ⬜️ Ignored (Inspect) | Visit Preview | Mar 1, 2024 11:46pm | |
| examples-vite-web | ⬜️ Ignored (Inspect) | Visit Preview | Mar 1, 2024 11:46pm |
![vercel[bot] avatar](https://avatars.githubusercontent.com/in/8329?v=4 "Published by a pub.dev verified publisher"){kind=small}
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
⚠️ CI failed ⚠️
The following steps have failed in CI:
- Turborepo integration tests
- TurboRepo Rust tests
See workflow summary for details
Would this be able to be used as a way to trigger tasks when files changes in dependencies/dependents of workspaces? A bit like what turbowatch allows to do but then natively.
The same infrastructure can definitely be used for alerting the client about package changes but this PR is scoped to relieving the client of having to calculate hashes and speeding up 'time to first task'. The streaming grpc api is just a stub for now.