New feature ideas

[yottahmd]

As we continue to maintain our project, we're always on the lookout for new features that can improve its usefulness for our users.

If you have an idea for a new feature, please add a comment to this issue with a brief description of what it would do and how it would benefit users. You can also include any relevant links or resources that might help us better understand your idea.

We can't guarantee that we'll be able to implement every feature suggestion we receive, but we're always grateful for your input and feedback. Thank you for helping us make our project even better!

To get things started, we've compiled a list of features that we're considering implementing.

• Implement a robust file database to store execution history data, replacing the current JSON file based system.
• Allow concurrent execution of DAGs, enabling users to run multiple instances at the same time.
• Add user management capabilities, giving users more control over authorization.
• Redesign the Web UI for a more intuitive user experience.
• Make it possible to use dagu as a Go module in other programs.
• Redesign the APIs and documentation
• Support multiple workers, allowing users to manage multiple servers in a single web server instance.
• Support for Slack integration.
• Support for cross-DAG dependencies to allow for more complex workflows.
• Improved documentation, including user guides, tutorials, and examples.
• Support standalone database
• Support FaaS execution (AWS, GCP, etc)?
• Handling Multiple Projects for DAGs directory and Web UI

[spreadsheetfun]

Any idea about storing the jobs/runs in a database like postgres...so that its more scalable and also reliable/fault tolerant.

[yottahmd]

Thank you for the wonderful suggestion! I would like to keep it self-contained so that users don't need to install database separately. One possible option would be SQLite. However, I think we can offer options for users to connect to a standalone Postgres database if they prefer.

[Arvintian]

About support integration with FaaS like aws Lambda?

Now, i use the http executor to do this.

[yottahmd]

@Arvintian That sounds useful, let's explore the way to handle it.

[stuartskelton]

Could there be an option to say `after this task wait X seconds before executing it the next step?

[yottahmd]

@stuartskelton Yes, currently, the delaySec option is available for DAG configuration as below. This could also be added to Step configuration, even for v1, such as delaySecOnFinish.

delaySec: 1                          # Interval seconds between steps

[dispensable]

Is it possible to add a Prometheus like metrics system to expose dags status ? Those metrics can be used to monitoring SLA miss or critical event during dag running.

[yottahmd]

@dispensable Sounds nice. Is it possible to see an example OSS with a similar feature? Let's look at it to consider our implementation.

[dispensable]

https://github.com/robinhood/airflow-prometheus-exporter

I haven't see single instance cron alternative combined with metrics. But there are many airflow prometheus exporter.

[sidneiweber]

Slack integration is interesting

[shbfy]

DAGs running other DAGs would be neat!