facebook
Change title of plot returned by `plot_marginal_effects` from "% better than experiment average" to "% higher..."
My metric is a "lower is better", and the "% better than experiment average" y-axis seems to be associated with a positively oriented scoring metric.
@sgbaird I might be misunderstanding your question but I believe y-axis plots relative improvement over the average.
When I had an objective that's being minimized, it seemed like it was actually plotting % worse than average rather than % better, but I could have been misinterpreting it.
@sgbaird, I don't think this behavior is expected –– if you wanted to add a screenshot of the plot you are seeing, that would be very helpful!
Haven't made a reproducer, but took a look at the functions: https://github.com/facebook/Ax/blob/57ba8714902ac218eb87dc2f90090678aa307a43/ax/plot/marginal_effects.py#L18-L69
From what I can tell, greater_is_better or equivalent isn't handled directly (just assumed) in plot_marginal_effects, and the only other two relevant functions are get_plot_data and marginal_effects which upon inspection don't seem to account for this either.
Simplest fix would be to change: https://github.com/facebook/Ax/blob/57ba8714902ac218eb87dc2f90090678aa307a43/ax/plot/marginal_effects.py#L66 to:
"title": "% higher than experiment average",
Since "higher" doesn't mean "better" to make it agnostic to maximization vs. minimization.
Here's an example of plot_data.as_dict() (using a breakpoint):
{'in_sample': {'0_0': PlotInSampleArm(name='0_0', parameters={'mu1_div_mu3': 1.5989093706011772, 'mu2_div_mu3': 46.885764732956886, 'std1_div_mu3': 85.80620403224603, 'std2_div_mu3': 6.035224910732358, 'std3_div_mu3': 93.63810631372034, 'comp1': 0.5191824799403548, 'comp2': 0.18046203907579184}, y={'vol_frac': 0.371}, y_hat={'vol_frac': 0.3727232216781629}, se={'vol_frac': nan}, se_hat={'vol_frac': 0.03455181048087161}, context_stratum=None),
'1_0': PlotInSampleArm(name='1_0', parameters={'mu1_div_mu3': 33.65488185826689, 'mu2_div_mu3': 42.546942168846726, 'std1_div_mu3': 49.672852043155586, 'std2_div_mu3': 96.46783315734938, 'std3_div_mu3': 12.900887220911683, 'comp1': 0.1950044333934784, 'comp2': 0.1061813235282898}, y={'vol_frac': 0.077}, y_hat={'vol_frac': 0.08716551069521344}, se={'vol_frac': nan}, se_hat={'vol_frac': 0.03510330756818702}, context_stratum=None),
'2_0': PlotInSampleArm(name='2_0', parameters={'mu1_div_mu3': 1.4598703954478836, 'mu2_div_mu3': 49.71556632665323, 'std1_div_mu3': 99.75099225102619, 'std2_div_mu3': 0.178245364423134, 'std3_div_mu3': 100.0, 'comp1': 0.9047762654495796, 'comp2': 0.05612461844249965}, y={'vol_frac': 0.486}, y_hat={'vol_frac': 0.4789201623711553}, se={'vol_frac': nan}, se_hat={'vol_frac': 0.034511312343398416}, context_stratum=None),
'3_0': PlotInSampleArm(name='3_0', parameters={'mu1_div_mu3': 1.0, 'mu2_div_mu3': 44.323065721122894, 'std1_div_mu3': 73.93810085027796, 'std2_div_mu3': 7.1356644632809925, 'std3_div_mu3': 91.55800266400693, 'comp1': 0.1569580900604541, 'comp2': 0.3070654231345528}, y={'vol_frac': 0.339}, y_hat={'vol_frac': 0.3374571764963387}, se={'vol_frac': nan}, se_hat={'vol_frac': 0.03481728876580751}, context_stratum=None),
'4_0': PlotInSampleArm(name='4_0', parameters={'mu1_div_mu3': 1.0, 'mu2_div_mu3': 42.437476484811775, 'std1_div_mu3': 92.06548435738938, 'std2_div_mu3': 0.10999988260144843, 'std3_div_mu3': 72.67951442423258, 'comp1': 0.9818462215312532, 'comp2': 0.0005357062858401542}, y={'vol_frac': 0.454}, y_hat={'vol_frac': 0.45073383688941293}, se={'vol_frac': nan}, se_hat={'vol_frac': 0.034679235819911794}, context_stratum=None)},
'metrics': ['vol_frac'],
'out_of_sample': {},
'status_quo_name': None}
and effect_table:
Name Level Beta SE
0 mu1_div_mu3 1.000000 13.650620 7.082151
1 mu1_div_mu3 1.459870 38.033832 9.947280
2 mu1_div_mu3 1.598909 7.425869 9.958769
3 mu1_div_mu3 33.654882 -74.877254 10.117456
4 mu2_div_mu3 42.437476 29.910001 9.995623
5 mu2_div_mu3 42.546942 -74.877254 10.117456
6 mu2_div_mu3 44.323066 -2.738471 10.035233
7 mu2_div_mu3 46.885765 7.425869 9.958769
8 mu2_div_mu3 49.715566 38.033832 9.947280
9 std1_div_mu3 49.672852 -74.877254 10.117456
10 std1_div_mu3 73.938101 -2.738471 10.035233
11 std1_div_mu3 85.806204 7.425869 9.958769
12 std1_div_mu3 92.065484 29.910001 9.995623
13 std1_div_mu3 99.750992 38.033832 9.947280
14 std2_div_mu3 0.110000 29.910001 9.995623
15 std2_div_mu3 0.178245 38.033832 9.947280
16 std2_div_mu3 6.035225 7.425869 9.958769
17 std2_div_mu3 7.135664 -2.738471 10.035233
18 std2_div_mu3 96.467833 -74.877254 10.117456
19 std3_div_mu3 12.900887 -74.877254 10.117456
20 std3_div_mu3 72.679514 29.910001 9.995623
21 std3_div_mu3 91.558003 -2.738471 10.035233
22 std3_div_mu3 93.638106 7.425869 9.958769
23 std3_div_mu3 100.000000 38.033832 9.947280
24 comp1 0.156958 -2.738471 10.035233
25 comp1 0.195004 -74.877254 10.117456
26 comp1 0.519182 7.425869 9.958769
27 comp1 0.904776 38.033832 9.947280
28 comp1 0.981846 29.910001 9.995623
29 comp2 0.000536 29.910001 9.995623
30 comp2 0.056125 38.033832 9.947280
31 comp2 0.106181 -74.877254 10.117456
32 comp2 0.180462 7.425869 9.958769
33 comp2 0.307065 -2.738471 10.035233
Which seems to confirm that it doesn't support directly "lower is better".
Of course, the easy workaround in the interrim is to update the yaxis title (see https://github.com/facebook/Ax/issues/726#issuecomment-1095927782 for to_plotly):
setup
import plotly.graph_objects as go
from ax.plot.marginal_effects import plot_marginal_effects
def to_plotly(axplotconfig):
data = axplotconfig[0]["data"]
layout = axplotconfig[0]["layout"]
fig = go.Figure({"data": data, "layout": layout})
return fig
axplotconfig = plot_marginal_effects(model, metric)
fig = to_plotly(fig)
update y title
fig.layout["yaxis"].update(
title_text="Percent higher than experimental average"
)