dagster
dagster copied to clipboard
[DOC-430] Add Managing Concurrency Guide
Adapted / edited the original doc for the new guide format.
I had a couple of questions as I was working on it:
- Should we keep the following sections or move them to a "concepts" page?
- glossary
- Setting priority for ops/assets
- Freeing concurrency slots
- Throttling concurrency-limited runs
- Troubleshooting
- tag_concurrency_limits could use a major look -- is bug noted here accurate, or does this still work with Postgres / MySQL?
- question: include Dagster insance settings (
dagster.yaml / Dagster+ Deployment settings) examples as inline YAML, keep the folder with the dagster.yaml? To repro, need to
export DAGSTER_HOME=$(pwd)/global_concurrency` (and for tag_concurrency, need an OSS version using Postgres / MYSQL) - (from Linear ticket) How can I set concurrency limits on a resource? -- I don't think this is possible, other than via using the concurrency tags.