dagster
dagster copied to clipboard
[cli] Implement `dagster definitions validate`
Summary & Motivation
This PR implements dagster definitions validate
, a command to validate the definitions of a Dagster project.
Expected behavior:
Have a command to validate the definitions of a Dagster project. A behavior similar to Definitions.validate_loadable is expected.
Example:
> dagster definitions validate -f path/to/my/definitions.py
Validating definitions in path/to/my/definitions.py
...
> dagster definitions validate -m my_dagster_project.definitions
Validating definitions in my_dagster_project.definitions.
...
> dagster definitions validate
Validating definitions in ...
...
Input:
Similar to dagster dev
as described here - the definitions can be validated:
- from a file, specified as a command line argument, eg.
definitions.py
- from a module, specified as a command line argument, eg.
my_dagster_project.definitions
- without command line arguments, read from
pyproject.toml
orworkspace.yml
.
Output:
Initially as logs in stdout/stderr.
In scope:
- Support the validation of Dagster definitions defined in a
Definitions
object. - Implement a behavior similar to
Definitions.validate_loadable
where the enclosed definitions will be loadable by Dagster:- No assets have conflicting keys.
- No jobs, sensors, or schedules have conflicting names.
- All asset jobs can be resolved.
- All resource requirements are satisfied.
Out of scope:
- Support the validation of Dagster definitions defined in a
Repository
object.- See comment here
- Validate definitions dependencies across code locations
- See thread here
- Could be added in a next iteration.
Potential improvement and next steps:
- Support usage in CI/CD
- Support output as JSON
- Support definitions constraints
- eg. Sensors'
minimum_interval_seconds
should always be of a at least 120 seconds.
- eg. Sensors'
- Support validation of definitions with dependencies across code locations.
How I Tested These Changes
Tested locally BK with additional tests