airflow icon indicating copy to clipboard operation
airflow copied to clipboard
verified publisher icon apache

AIP-38 | Migrate to new UI

Open bbovenzi opened this issue 1 year ago • 4 comments

Meta epic to track all legacy UI pages we need to migrate over to the new UI

See here for designs: https://github.com/apache/airflow/issues/42371

I'll probably update the main issue description. But here's the priority for the month:

December core user stories

  • Dags List
    • [x] Dag Card to show last and next runs and a chart of recent runs
    • [x] Can pause/unpause a Dag from the list
    • [x] Can trigger a dag from the list #44857
    • [x] Filter dags by last dag run state, paused, and tags
    • [x] Sort dag list by display name, latest run, next run,
    • [x] Search for a dag by dag_id or dag_display_name
    • [ ] Show dag import errors on dag list level
  • [ ] Dag Details
    • [x] Click on a row in the Dags list to open up Dag Details
    • [x] Show header that has the same information as the Dag Card and includes actions to:
      • [x] Trigger basic Dag #44857
      • [ ] Use dag params to generate forms for triggering a dag #44858
    • [x] List all dag runs and link to a specific Dag Run
    • [x] List all tasks
      • [ ] Include task group information
    • [x] Render Dag Code
  • [ ] Dag Visualizations
    • [x] Graph
      • [x] Show the nodes and edges of a graph
      • [ ] Toggle showing the external dependencies of a graph (sensors, triggers, data assets with conditions)
      • [ ] Show a specific dag run in the graph with each node filled with task instance or asset event info
      • [ ] Click on a task to select it as a task or task instance
    • [ ] Grid
      • [ ] Explore use of icons for each state to be more accessible. And how to balance that with run_type, try_number and has a note functionality
    • [ ] Gantt
    • [ ] Dropdown selector on each viz to change between dag runs
    • [ ] Side drawer on each viz to see details about a task instance or dag run and the ability to quickly clear or mark as a failure
    • [ ] Auto refresh grid data to see dags running
  • [x] Task Details
    • [x] List all task instances
  • [x] Dag Run Details
    • [x] List all task instances
    • [ ] Manually mark dag run as failed or success
    • [x] Clear a dag run
  • [ ] Task Instance Details
    • [ ] Manually mark a task failed or success
    • [ ] Clear a task instance #44860
    • [x] See task instance logs across tries #44663
    • [ ] See task instance details across tries
    • [ ] List all mapped task instances
      • [ ] Allow selecting a single mapped instance with all task instance functionality
    • [x] See task instance xcoms #44667

January

  • [ ] Backfills
    • [ ] Trigger a backfill for a dag and see how many runs will be created before finally submitting
    • [ ] See if a dag has active backfills and see the state of all the backfilled runs
  • [ ] Versioning
    • [ ] Show version on dag/dagrun/task/task instance
    • [ ] When viewing an older dag run, fetch the dag code for that version
    • [ ] Fetch the dag structure for an older version or give an alert that the structure has changed

Dashboard

Import Errors https://github.com/apache/airflow/issues/43711 Pool Summary #43328 Historic Dag Run metrics #42700 Quick links to DAGs List #44673 Add an event timeline to help with date range selection Summary of components ( scheduler(s) , celery-worker(s) , triggerer(s) , dag-processor(s) )

Dags List

Search by display_name #42714 Filter by tags #43712 Filter by last run state #42715 Update sort options #43043

Dag Details

Trigger Versioning (to scope) Overview List Runs List Tasks Grid View #44671 Graph View #44200 Gantt Chart #44672 Code #43713 Events per Dag #43705

Backfills

List backfills https://github.com/apache/airflow/issues/43967 Active backfills banner https://github.com/apache/airflow/issues/43968 Backfill details Create a new backfill https://github.com/apache/airflow/issues/43969 Pause/cancel backfill

Dag Run Details

Header Card Clear Run Mark run as... Change Dag Run Note

Task Details https://github.com/apache/airflow/issues/44669

Header Card Overview List Task Instances

Task Instance Details

Header Card Logs https://github.com/apache/airflow/issues/44663 XCom https://github.com/apache/airflow/issues/44667 Rendered templates Clear TI Mark TI as... Change TI Note

Assets

To be scoped

Admin

Connections

List Connections #43703 Edit Connection Delete Connection Test Connection

Events

List Events #43704 Filter events

Pools

List Pools #43706 Edit Pool Delete Pool Create Pool

Providers

List Providers #43708

Variables

List Variables #43709 Add Variable Edit Variable Delete Variable

User

Light/Dark toggle #42823 #11334 Timezone selection #42817 Color customization #43054

Plugins

New React UI Plugins #42702 Iframe Plugins #42708

Config

Remove unnneeded config #43519 Render UI with config values

bbovenzi avatar Nov 05 '24 17:11 bbovenzi

Filter by tags https://github.com/apache/airflow/pull/43715

This appears to link to something else.

hterik avatar Nov 25 '24 07:11 hterik

Filter by tags #43715

This appears to link to something else.

Good catch. Fixed.

bbovenzi avatar Nov 25 '24 15:11 bbovenzi

I feel like we have everything from the backend perspective. (or planned such as asset dependencies).

The only points I'm not so sure are:

  • Include task group information (I'm not sure we have this)
  • Show dag import errors on dag list level (I am not sure how we want to display them, as in the legacy we can put them at the top of the dag list, but I don't think we can group them by dag_id there's not really a relation there in the db model)

pierrejeambrun avatar Dec 12 '24 09:12 pierrejeambrun

I feel like we have everything from the backend perspective. (or planned such as asset dependencies).

The only points I'm not so sure are:

  • Include task group information (I'm not sure we have this)
  • Show dag import errors on dag list level (I am not sure how we want to display them, as in the legacy we can put them at the top of the dag list, but I don't think we can group them by dag_id there's not really a relation there in the db model)
  • I think almost all Task Group information is handled by grid_data and structure endpoints
  • We literally just need to copy the component from the home page and also show it at the top of the page on the dags list

bbovenzi avatar Dec 12 '24 16:12 bbovenzi

I'll work on the mark as.... (failed/success) etc. For dagruns and tasks.

pierrejeambrun avatar Jan 21 '25 15:01 pierrejeambrun

@bbovenzi Not sure if this is already planned, but could we add a conf column to the DAGs run UI page like in Airflow 2?

aryavj1 avatar Apr 19 '25 20:04 aryavj1

@aryavj1 Do you mind opening a separate github issue for that? I think we can look into how to use the "expanding" feature in react-table to quickly reveal the config for a dag run or the extra for an asset event

bbovenzi avatar Apr 22 '25 15:04 bbovenzi

Airflow 3 is out. Closing this meta issue, anything else should be its own feature request.

bbovenzi avatar Apr 22 '25 17:04 bbovenzi

@bbovenzi I have opened an issue here https://github.com/apache/airflow/issues/50912

aryavj1 avatar May 21 '25 16:05 aryavj1