dagster icon indicating copy to clipboard operation
dagster copied to clipboard

[DOC-430] Add Managing Concurrency Guide

Open cnolanminich opened this issue 5 months ago • 10 comments

Adapted / edited the original doc for the new guide format.

I had a couple of questions as I was working on it:

  • Should we keep the following sections or move them to a "concepts" page?
    • glossary
    • Setting priority for ops/assets
    • Freeing concurrency slots
    • Throttling concurrency-limited runs
    • Troubleshooting
  • tag_concurrency_limits could use a major look -- is bug noted here accurate, or does this still work with Postgres / MySQL?
  • question: include Dagster insance settings (dagster.yaml / Dagster+ Deployment settings) examples as inline YAML, keep the folder with the dagster.yaml? To repro, need to export DAGSTER_HOME=$(pwd)/global_concurrency` (and for tag_concurrency, need an OSS version using Postgres / MYSQL)
  • (from Linear ticket) How can I set concurrency limits on a resource? -- I don't think this is possible, other than via using the concurrency tags.

cnolanminich avatar Sep 06 '24 19:09 cnolanminich