hamilton icon indicating copy to clipboard operation
hamilton copied to clipboard
verified publisher icon DAGWorks-Inc

Enhanced viz for pyspark

Open elijahbenizzy opened this issue 2 years ago • 0 comments

Is your feature request related to a problem? Please describe. We have two types of dependencies:

  1. Logical (how the DAG was defined)
  2. Physical (how it is executed)

Currently we think of these as the same, but the viz is ugly. See @roelbertens comment here: https://github.com/DAGWorks-Inc/hamilton/pull/309#issuecomment-1701614097.

Describe the solution you'd like

  • The logical dependencies should shape the DAG.
  • The physical ones should be marked differently.
  • Perhaps this should be adjustable

Describe alternatives you've considered Moving it to visualization within h_spark so we don't have to generalize

Additional context

See line from darkshore_likelihood to durotar_likelyhood here: image

elijahbenizzy avatar Aug 31 '23 23:08 elijahbenizzy