Issues with Waterfall Plot customizability

[OSuwaidi]

I really love the newly introduced waterfall plot, it's super practical and insightful.

However, I faced a couple of issues when trying to customize it into a funnel plot:

1. It doesn't support adding labels like other plots do. Here is a MRE:

from lets_plot import *
from lets_plot.bistro.waterfall import *
import polars as pl
LetsPlot.setup_html()
stages = ('Initiated Registration', 'B4 Redirection', 'Reached Redirection', '@ Redirection', 'Redirected', 'After Redirection', 'Total Registered')
data = {
    'stage': stages,
    'count': [598, -264, 334, -4, 330, -85, 245],
    'measure': ['absolute', 'relative', 'total', 'relative', 'total', 'relative', 'total'],
}
df = pl.DataFrame(data)
# Add each step's/stage's percentage of initial count:
df = df.with_columns(
    ((pl.col('count')/pl.lit(598.) * pl.lit(100.)).abs().round()
    .cast(pl.String) + pl.lit("%"))
    .alias('pct')
    )

water = (waterfall_plot(df, 'stage', 'count', measure='measure', label='pct')  # doesn't display percentages
 + scale_y_continuous(breaks=list(range(0, 601, 100)))
 + coord_cartesian(ylim=(0, 650))
)

I also tried label=layer_labels().line("@pct") but that didn't work either. I understand that by default, the plot itself layers a label showing bar's value, but allowing to show percentages (either based on initial or previous) would be even more insightful for the plot (similar to how plotly has textinfo = "value+percent initial/previous" in their funnel plots.) And that would be relevant to any generic waterplot (% changes is very important).

So, I tried to manually add the labels using geom_text, which brings us to issue number 2: 2. It doesn't allow layering different geom objects on it. MRE:

df = df.with_columns(pl.col("count").abs().alias("abs_count"))
water + geom_text(
    aes(x=as_discrete('stage'), y='abs_count', label='pct'), data=df, color='black', va='center', hjust='left',
 )

That didn't change anything. I tried adding ggplot() to the beginning, but it just showed the geom text object, and adding it to the end changed nothing either. (P.S. I know that the height I used to show the text (abs_count) is incorrect for relative values because their heights would be previous height + their signed value // 2 (to get midpoint). Please add this feature lol.)

3. Lastly, it doesn't enable customizing the bar fill colors (other than presets). MRE:

df = df.with_columns(
    (pl.when(pl.col("count") >  pl.lit(0)).then(pl.lit("green")).otherwise(pl.lit("red"))).alias("color")
)
 waterfall_plot(df, 'stage', 'count', measure='measure', label='pct', fill='color')  # gives error: can't convert

And I also tried fill=aes(fill='color') but that didn't work either. It would be really nice to support a color mapping like this for example: {"total": "green", "increase": "blue", "decrease": "orange"}

P.S. Also while we're at it, the hjust parameter in geom_text feels so glitchy, because while the docs says that it accepts a number between 0 and 1, it only reacts (changes) when its either 0 or 1 (not for values in between). Additionally, setting it to left moves text to the right, and vice-versa, which is confusing.

[alshan]

Hi!

It doesn't support adding labels like other plots do.

We are planning to support layer labels (a.k.a. annotations) in future but for now labels are firmly linked to the "y" parameter. If however the "y" variable carries percentile value, then should be able to add '%' suffix to the labels:

  waterfall_plot(df, 'stage', 'pct', measure='measure',
                 label_format="{}%", label=element_text(color="black"),
...

Note, the "label" parameter is for styling only.

And I also tried fill=aes(fill='color') but that didn't work either. It would be really nice to support a color mapping like this for example: {"total": "green", "increase": "blue", "decrease": "orange"}

To assign different color by the "measure" value try to use manual scale:

 + scale_fill_manual(values={"total": "green", "increase": "blue", "decrease": "orange"})

the hjust parameter ...

Thanks, we will take a look.

[OSuwaidi]

Hey @alshan,

Thanks for your response. I believe that scale_fill_manual (and scale_color_manual) are buggy, at least when using them on a waterplot.

Here's what I got when I added your suggestion to the waterplot mentioned above:

water + scale_fill_manual(values={"total": "green", "increase": "blue", "decrease": "orange", "absolute": "red"})

Which shows that there's an issue with the values-to-aesthetics mapping, because it seems that:

1. "total" is mapped to "absolute"
2. "increase" is mapped to "decrease"
3. "decrease" is mapped to "total"
4. "absolute" is mapped to "increase" (maybe?)

To verify, I tried to add + scale_color_manual(values={"total": "green", "increase": "blue", "decrease": "orange", "absolute": "red"}) to the example in the docs, and I got: Which verifies the inconsistency is the values-aesthetics mapping.

Also, regarding:

2. It doesn't allow layering different geom objects on it.

Is this the intended behavior for this kind of plot?

[alshan]

To verify, I tried to add + scale_color_manual(values={"total": "green", "increase": "blue", "decrease": "orange", "absolute": "red"}) to the example in the docs, and I got:

Maybe but to be honest, the legend looks correct. Could you double check if you have correct "measure" values in df? Based on what I can see in the chart, it looks like your 'measure' column might contain values like:

 'measure': ['total', 'total', 'increase', 'total', 'total', 'decrease']

It doesn't allow layering different geom objects on it. Is this the intebded behavior for this kind of plot?

Yes, it is the intended behavior, but it looks like this can be improved to support additional geom layers. We will take a look at it.

[OSuwaidi]

Hey @alshan,

Yes, the legend is indeed mapped correctly but the laid out aesthetics are clearly mismatched. To clarify, I did not create a dataframe or mess around with the data from the docs, I took it as it is and just added the scale_color_manual:

import numpy as np
from lets_plot import *
from lets_plot.bistro.waterfall import *
LetsPlot.setup_html()
n, m = 10, 5
categories = list(range(n))
np.random.seed(42)
data = {
    'x': categories,
    'y': np.random.randint(2 * m + 1, size=len(categories)) - m
}
waterfall_plot(data, 'x', 'y', \
               threshold=2, \
               width=.7, size=1, fill="white", color='flow_type', \
               hline=element_line(linetype='solid'), hline_ontop=False, \
               connector=element_line(linetype='dotted'), \
               label=element_text(color='flow_type'), \
               total_title="Result", show_legend=True) + scale_color_manual(values={"total": "green", "increase": "blue", "decrease": "orange", "absolute": "red"})

Which gives the mismapped values-aesthetics plot above. Similarly, from my example, my dataframe had the following measure column:

'measure': ['absolute', 'relative', 'total', 'relative', 'total', 'relative', 'total']

But the fill colors look like there were mapped to: ['total', 'increase', 'decrease', 'increase', 'decrease', 'increase', 'decrease'], hence the inconsistency.

[alshan]

I did not create a dataframe or mess around with the data from the docs, I took it as it is and just added the scale_color_manual

I see, in this case "measure" should have capitalized 1st letter:

+ scale_fill_manual(values={"Total": "green", "Increase": "blue", "Decrease": "orange", "Absolute": "red"})

sorry for confusion, we'll try to improve the docs.

[alshan]

Hi @OSuwaidi, I hope you are doing well. In the latest v4.7.0 we provide options for configuring custom labels on bars (via relative_labels and absolute_labels parameters) as well as support for additional geometry layers in waterfall plots.

I hope this addresses your concerns well.