Animating layout and data simultaneously

[rreusser]

One feature missing in animation is simultaneous data and layout transitions. The docs note that this is explicitly disabled. Closely related is that I'm not actually aware of a library that gets things strictly correct. Highcharts, for example, will transition both simultaneously, but in screen space, not data space. That means the data doesn't match the axis as it transitions. On top of that, to get this working correctly, instantaneous updates and finite transitions and ui interactions basically all have to be carefully coordinated to get this right.

Opening this issue to collect examples and track progress on improving the robustness.

[alexcjohnson]

One concern I have is the implicit dependence between data and layout through autorange, and how it seems like currently that could leave you in an inconsistent state:

• set up an autoranged plot
• do an animation that changes the data (and changes what the autorange results should be)
• but the autorange won't actually reevaluate, so the plot isn't really in the state implied by data and layout, until some other event causes a full redraw.

[rreusser]

Could that be controlled via an animation config flag so that the default result is consistent and you have the option to augment it with fancier behavior?

[etpinard]

cc @antoinerg

[nicolaskruchten]

I would really like to understand more about what's blocking us from currently doing this :)

@rreusser the general feeling on the team today is something like "Ricky looked at this for a while and thought it was really hard to do right so we're stuck" but I find this a bit unsatisfying... do you happen to recall anything about what the sticking points were? Were there some pathological edge cases?

I'm not sure I understand this:

Highcharts, for example, will transition both simultaneously, but in screen space, not data space. That means the data doesn't match the axis as it transitions.

[nicolaskruchten]

(I've added this to the 1.52.0 milestone not as a "we have to completely implement this in 1.52.0" but during this dev cycle I'd like us to gain clarity on what can/can't work and what would be needed)

[rreusser]

@nicolaskruchten If I recall correctly, there were a couple issues.

The first issue was simply state management since plotly generally is eager to tear things down and redraw to ensure consistency. If I recall correctly, I found it very challenging to manage the axes objects and figure out which was current or merely an outdated reference.

Regarding the specific details of animation, perhaps negligible is that the intermediate states are tweened but not strictly mathematically valid since it transitions in screen-space rather than in data-space. For example, imagine transitioning the axes and the data at the same time. The axis transition can be accomplished via a transform which would take the data with it (rather than tearing down the plot and redrawing on every frame), but the data is transitioned in screen space. The result might be that the start and end states are correct and things in between are smooth but that the data doesn't track correctly with the axes between the two states.

That's not all that undesirable since it looks smooth, but here's where it got very difficult: When you drag the plot, it doesn't redraw things on every mouse event. If I recall correctly, it applies a transform and then when you mouseup it tears down the plot and redraws. This, I think, was the fundamental source of difficulty since the task is state management combined with two animation mechanisms: one mechanism is a requestAnimationFrame loop that updates an ephemeral axis-dragging sort of transform, and the other is browser svg transitions. I could definitely be convinced that perhaps a more promising pathway than the request animation frame loop is to use a matrix with browser transitions to transform the axes.

That's a few thoughts based on my recollections, but I'm glad to clarify and discuss further! I really tried to get it working and felt like it was very close, but I never quite was able to overcome the state management concerns and get things working well.

[alexcjohnson]

Thanks @rreusser for the extra context!

A concrete example of screen-space vs data-space transitions: Let's say you have a transition that both doubles the size of a bar, and halves the span of the axis it's drawn on. And let's further simplify to say the transition is linear. As a result the bar should end up 4x bigger than it started. If we do the entire transition in screen space, the bar will linearly expand, so halfway through it will be 2.5x its original size. But done in data space, the bar would grow linearly while the axis is zooming linearly, so there would be a quadratic component to the bar's size trajectory. Halfway through, it would be 1.5x * 1.5x = 2.25x its original size.

I'm sure there are cases where the distinction is bigger than that; but seems to me in general even if it's not 100% accurate, as long as all the pieces make the same choice we should be fine with screen-space transitions. Bars that are supposed to be connected to each other will be connected to each other throughout the transition; points that are supposed to switch positions will still switch positions, just perhaps not at the same fraction of the overall transition.

[rreusser]

One other thing to consider is log axes, which I think are similarly tolerable in screen space and as long as it's not require that data tracks the axes.

And just for the sake of argument, I don't know anything about the particular product requirements here, but I will note that the OpenGL approach is generally to use a model matrix and (of course) completely clear and redraw the entire plot on every frame. I don't know if it's possible or reasonable or feasible, but there is a possibility that this would actually be a comparatively easy thing to accomplish in WebGL as opposed to SVG which prefers the statefulness and persistence which make this difficult.

[rreusser]

Ah, now that I think about it, layout is easy in WebGL, but d3's management of state is what makes data transitions easy. Sigh.

[nicolaskruchten]

I don’t really follow this screen/data argument... “during a transition” the state of a vis is never “accurate” in the sense that if on my way from 2 to 3 (data) I always travel “through” 2.5 even though that’s not data per se.

[nicolaskruchten]

With respect to dragging, @rreusser are you describing problems that would occur when dragging a plot during a transition?

[nicolaskruchten]

Is the screen/data thing just that a “linear” transition only applies separately to the axis and the bar but not their relationship @alexcjohnson ?

[rreusser]

I don’t really follow this screen/data argument... “during a transition” the state of a vis is never “accurate” in the sense that if on my way from 2 to 3 (data) I always travel “through” 2.5 even though that’s not data per se.

In particular I mean that data is subject to the layout, but SVG transitions mean that data really just transitions in screen space rather than transitioning under the layout, which itself is transitioning. I'm not sure if that's clear yet. Instead, the structure leads to layout and the data transitioning independently to correct start and end states, but they're not truly consistent in intermediate states. In my head, I'm picturing a hierarchy in which data should live under the layout and move accordingly, but instead it's more of a situation where data and layout transition independently, like siblings—to correct states, of course, but still independently.

With respect to dragging, @rreusser are you describing problems that would occur when dragging a plot during a transition?

~Yes, and if I recall correctly, layout animation uses some similar principles or even just piggybacks on the same machinery to update the view without replotting and recomputing the whole SVG hierarchy.~

Edit: Sorry, yes that is a thing and that's why dragging was disabled during transitions, but that's basically the issue. By piggybacking on the same machinery, it's hard to use it for both at the same time, though probably very possible and not fundamentally difficult.

[nicolaskruchten]

Ok thanks for the clarifications!

I am totally fine with data and layout independently transitioning to their final destinations and their relationships therefore being ambiguous/wrong in between :)

I’m also totally fine with disallowing interaction during transitions :)

Under those relaxed assumptions, are there any known blockers or pitfalls we should look out for?

[rreusser]

I think the biggest challenges were simply managing the corner cases, async state. Some of that was definitely my shortcomings. I think single thing that caused me the most trouble was axes instances before and after supply defaults since, as mentioned, I had a lot of trouble figuring out which things were current and which were appropriate to hold onto and use at which time asynchronously. The other big thing was accidental double-transforms since it tends to fall into one of these:

1. if you're only changing the layout, then the ephemeral drag-transform (I forgot specifically what that mechanism is called) is easy and very fast and doesn't require replotting the data until it's complete, at which time you can replot to ensure consistency
2. if you're only changing the data, then you can use d3 transitions and let the trace do its work
3. however, if you're applying both, then combining the above two strategies will double-transform the data in the sense that it will get ephemeral-drag-matrix-transformed but also svg-transitioned. I don't absolutely know the best strategy here, but it's the sort of issue that had me pulling my hair out a bit. I would think it had worked, but I don't think I had a fully coherent strategy for managing this interaction and so I really had to fight it and eventually had to opt for either animating one or the other but not both.

I would love to see this all working smoothly, so I'm glad to continue offering thoughts/experiences or just helping work through difficult bits of my code. It was pretty challenging so it's likely there are some bits that might need explaining. Don't hesitate to ask.

[nicolaskruchten]

Thanks very much for being willing to help out so long after the fact 🙏

In the case where both get applied and the data gets double-transformed, this is when both layout and data get to their final destinations "in the own way" right? Or do they just not make it to the right place, or take a weird path or...?

I'm having trouble concretizing what you're saying into a case of "when I do X I get Y outcome which is bad because of Z" :)

Like in a case where I have a scatter plot transition from state A to state B and in state B the markers are meant to be in a place that's not currently in range, is the problem that:

1. "when we transform both, scatter points don't end up in the right place with respect to axes"; or
2. "when we transform both, scatter points take a loopy path to their final destination which is correct"; or
3. "when we transform both, everything takes a straight-line path to its final destination, but the intermediate frames make no sense visually"; or
4. something else I'm not thinking of?

Put another way: are we having trouble doing what other libraries (i.e. Highcharts) do, or are we trying to do something "better" in some way and stuck at that point?

[rreusser]

Sorry this is vague and hand-wavey. I think I'm having trouble remembering the precise state the corner cases ended up in because I eventually had to pull the plug and cut out some feature interactions.

The double-transform issue was a case where things did not end up in the right state and were unacceptably buggy. To clarify, what I wanted was:

1. data transitions are handled by re-running the scatter trace plotting code and letting svg transitions do their magic in screen-space 👍
2. layout transitions are handled by applying a matrix to the traces as a whole and not re-running the scatter trace plotting code 👍

To help explain the double-transition issue, consider the following:

1. say a point goes from dataStart=[0, 0] to dataEnd=[1, 1]
2. say the layout zooms by a factor of two (zoomStart = 1, then zoomEnd = 2)

If you combine the strategies, you have to be very careful. If you truly handle data and layout transitions as completely separate, then perhaps you actually want to animate the data from zoomStart * dataStart to zoomStart * dataEnd and then apply the zooming as a completely separate step by transforming the scatter trace as a whole, after the plotting.

Or maybe you animate from zoomStart * dataStart to zoomEnd * dataEnd and don't transform traces with a matrix at all (this is perhaps preferable. see bottom paragraph for pros/cons. I think this is what Highcharts does)

If I'm being honest, I think the challenge is perhaps path-dependent development where piggybacking on the machinery already in place for static traces and mouse interactions was performant but not designed from the ground up to minimize feature interactions.

If I recall correctly, I think my requestAnimationFrame loop for animating layout does two things: it animates the axes and also applies a transform to the data. Perhaps it shouldn't apply that transform to the data at all in order to move it with the axes and it should instead replot all the traces with d3 transitions in order to trigger SVG transitions and get things into their correct state. As mentioned, then downside is that the data won't track the axes perfectly along the way, though it's probably much less complicated. I believe this is what highcharts does.

[rreusser]

To try to boil that down more, transforming traces with a transform matrix gets them to follow the axes perfectly, performantly, and correctly. But if you transition the data along the way, you have to consider that you're transitioning data in the coordinate system of where things started since the axis transition is applied at the end. So in that sense, the first option is that you could transition the data all relative to the original axes, then separate the changing axes into a matrix handled at the end. Restating this, it doesn't seem as bad?

This option, written as functions:

1. start at plot(initialTraceState, initialAxisState)
2. transition to plot(finalTraceState, initialAxisState)
3. apply a matrix that transform the traces along with the axes from initialAxisState to finalAxisState

The other option (the highcharts option, possibly?) is not to apply that matrix to the trace to get it to track the axes correctly but to instead to transition from the first state in screen-space to the final state, also in screen-space.

This option, written as functions:

1. start at plot(initialTraceState, initialAxisState)
2. transition to plot(finalTraceState, finalAxisState)
3. transitioning the axes is separate and unrelated to the traces, but things end up matching at the end.

I think I had double transforms because I had trouble tracking the axis objects and combined the two incorrectly:

1. start at plot(initialTraceState, initialAxisState)
2. transition to plot(finalTraceState, finalAxisState)
3. apply a matrix that transform the traces along with the axes from initialAxisState to finalAxisState. Things now end up double-transformed because we already tried to make it track the axes in step 2 by using the final axis state.

[nicolaskruchten]

Ahh, I see. I feel like doing things in the "possibly highcharts" option is conceptually easier and nice to look at, if less theoretically pure, let's say. Doing things in two disjoint transition steps would be better than the status quo, which is "transition one, snap the other, choose the order", but clearly having some way of simultaneously transitioning them would be nice.

[nicolaskruchten]

@alexcjohnson I'm having trouble understanding the test case you're outlining with "Let's say you have a transition that both doubles the size of a bar, and halves the span of the axis it's drawn on."

Is this like transitioning from:

1. a bar at x=1 with y=1, with y_range=[0-4] mapped onto 120px, so the bar is 30px tall
2. the same bar at x=1 with y=2, with y_range=[0-2] mapped onto the same 120px, so the bar is now 120px tall

In screen coordinates, linearly growing the bar to its destination means that halfway through the transition, the bar is at 75px, and the y_range is now mapping [0-3], meaning if you froze the frame and read off the bar-height it would be at 3*75/120=1.875 "incorrect data units" (it should be 1.5!)

If you did this in data coordinates, linearly growing the bar to its destination would mean that halfway through, it would be at 1.5 data-units, and therefore its height would be halfway along the axis which now maps [0-3] to 120px, so it should be at 60px.

Is this what you're saying? If so, should we really consider this a blocker?

[alexcjohnson]

As I said:

we should be fine with screen-space transitions

not a blocker

[nicolaskruchten]

OK cool. I couldn't replicate your "2.25" from above so I was just checking to make sure I wasn't subtly misunderstanding.

So given that, is it just a matter of "doing both the things we currently do simultaneously" and therefore is an easy lift to start or... ?

[nicolaskruchten]

PS I’m looking forward to being one step closer to Plotly.js being able to make bar chart races :P https://flourish.studio/2019/03/21/bar-chart-race/

[etpinard]

For @archmoj , a good way to "get into" the transition/animation code would be to look at some recent PRs and the transition_test.js suite.

• https://github.com/plotly/plotly.js/pull/3217 added the transition behaviour to Plotly.react. To do so, I had to write up a new version of the Plots.transition routine that that assumes that gd._fullData / gd._fullLayout is the "new" state (N.B. Plotly.animate assumes that gd._fullData / gd_fullLayout is the "old" state). https://github.com/plotly/plotly.js/pull/4167 made a fixup of that new code.
• https://github.com/plotly/plotly.js/pull/4262 fixed an issue related to the ordering for which axes and traces are transitioned

[etpinard]

(Other then that, I don't have much to add on top of what @rreusser has written down)

[jackparmer]

This issue has been tagged with NEEDS SPON$OR

A community PR for this feature would certainly be welcome, but our experience is deeper features like this are difficult to complete without the Plotly maintainers leading the effort.

Sponsorship range: $30k-$35k

What Sponsorship includes:

• Completion of this feature to the Sponsor's satisfaction, in a manner coherent with the rest of the Plotly.js library and API
• Tests for this feature
• Long-term support (continued support of this feature in the latest version of Plotly.js)
• Documentation at plotly.com/javascript
• Possibility of integrating this feature with Plotly Graphing Libraries (Python, R, F#, Julia, MATLAB, etc)
• Possibility of integrating this feature with Dash
• Feature announcement on community.plotly.com with shout out to Sponsor (or can remain anonymous)
• Gratification of advancing the world's most downloaded, interactive scientific graphing libraries (>50M downloads across supported languages)

Please include the link to this issue when contacting us to discuss.

[1001Josias]

PS Estou ansioso para estar um passo mais perto de Plotly.js poder fazer corridas de gráficos de barras :P https://flourish.studio/2019/03/21/bar-chart-race/

@nicolaskruchten, do you know if it is already possible to create bar chart race and export to video?