Incongruent Response Curves and Contribution Calculations in terms of ROI

[xijianlim]

Discussed in https://github.com/google/lightweight_mmm/discussions/49

^{Originally posted by xijianlim August 1, 2022} Hi all, this is something I've been noticing the provided code base using the adstock-hill . The response curves (in terms of ROI) do not match the contribution/cost ratios in the contribution dataframe outputs. This is one example:

As you can see, the ROI via the function plot.create_media_baseline_contribution_df will provide a ROI 11.9 (Contribution from the training set is 2820, impressions is 9468 and an average price of 0.025 would mean $236 for cost, yielding $11.9).

However, the Response curves clearly show a ROI well below 1 and hitting its diminishing profile.

To help emphasize this discrepancy, ive amended the code to return a dataframe from "plot.plot_response_curves"

This example was found using the util functions to generate the data.

[pabloduque0]

Hello @xijianlim ! Thanks for reporting this one!

There are a few small nuances that could lead to some difference in response curves vs contribution calculation. Although the difference that you report here seem bigger than I would expect.

I see that you are currently running a geo model. Could you confirm if you see this same issue on a national model? Just so I can rule some things out.

Did you use extra features for this model? If so, what are the coefficients of it?

Thanks

[xijianlim]

Hello again @pabloduque0 . Yes the same thing happens at the national model, and yes there were extra features in the model.

you can clearly see that channel_0's ROI profile is well below $1:

If I may offer an idea for the Response Curves:

instead of using the media ranges and constructing them using the mmm.predict function, would it be better if you feed them through the adstock-hillfunction equation but using the coefficients from the model?

something like this:

def media_transform_hill_adstock(media_mix_model,
                                  media_data,
                           lag_weight,
                           half_max_effective_concentration,
                           slope, apply_adstock, normalise):
ransforms the input data with the adstock and hill functions.

  Args:
  media_data: Media data to be transformed. It is expected to have 2 dims for
    national models and 3 for geo models.
  normalise: Whether to normalise the output values.

  Returns:
  The transformed media data.
  """
  if media_mix_model.n_geos > 1:
      lag_weight=jnp.repeat(lag_weight,media_mix_model.n_geos).reshape(half_max_effective_concentration.shape)
      slope = jnp.squeeze(jnp.repeat(slope,media_mix_model.n_geos).reshape(half_max_effective_concentration.shape))

  
  if apply_adstock:
    return media_transforms.hill(
    data=media_transforms.adstock(
        data=media_data, lag_weight=lag_weight, normalise=normalise),
    half_max_effective_concentration=half_max_effective_concentration,
    slope=slope)

  else:
    return media_transforms.hill(
    data=media_data,
    half_max_effective_concentration=half_max_effective_concentration,
    slope=slope)

percentage_add=2
media_mix_model=mmm
steps=25
prices=average_cost_per_media
media = media_mix_model.media
media_maxes = media.max(axis=0) * (1 + percentage_add)
half_max_effective_concentration=media_mix_model._mcmc.get_samples()['half_max_effective_concentration']#.mean(axis=0)
lag_weight=media_mix_model._mcmc.get_samples()['lag_weight']#.mean(axis=0)
slope=media_mix_model._mcmc.get_samples()['slope']#.mean(axis=0)
beta_media=media_mix_model.trace['beta_media'].mean(axis=0)

media_ranges = jnp.expand_dims(
      jnp.linspace(start=0, stop=media_maxes, num=steps), axis=0)

media_ranges=jnp.repeat(media_ranges,len(beta_media)).reshape(steps,len(beta_media),media_mix_model.n_media_channels, media_mix_model.n_geos)


media_response=beta_media*media_transform_hill_adstock(media_mix_model,
                                  media_ranges,
                           lag_weight,
                           half_max_effective_concentration=half_max_effective_concentration,
                           slope=slope, normalise=True)

[xijianlim]

I can confirm that constructing the response curves using the trace parameters (beta_media, lag_weight, half_max_effective_concentration and slop) as per above has resolved the issue. I've modified this in my version of the code base. I thought I'd share this with you as this has closer alignment with the media contributions. I suggest amending the response_curve plots with this method.

[pabloduque0]

Hello @xijianlim !

Thanks for your response.

There are a couple reasons why those two are designed differently. Mainly the response curves are a forward looking and the other approach is strictly backward looking. However for that we need to make a couple assumptions about lagging and extra features that might not be what users want in some scenarios. I agree that we should offer the flexibility to not take into account those assumptions, or make them optional or similar.

Let me looks into it a little bit and get back on this one.

[steven-struglia]

@xijianlim How did you go about plotting the response curves on the spend vs. KPI axis? I can't seem to make the dimensions or values work for my use case using your function above.

[MuhammedTech]

were you be able to plot it?

[xijianlim]

Hi, sorry for the delay in replying. To be clear, this is only for ADSTOCK-BETAHILL modelling. Unfortunately I can't share the exact code because it is now my company's PI but i'll demonstrate how you can plot this.

1. you have import this function from media_transforms.py

media_transforms.hill( data=media_transforms.adstock( data=media_data, lag_weight=lag_weight, normalise=normalise), half_max_effective_concentration=half_max_effective_concentration, slope=slope)

This is in effect is the "response curves" function. All that is needed is to use the trained models's "trace" functions to get the right parameters, namely the lagweight, slope, beta_media and half_effective concentration.

percentage_add=2
media_mix_model=mmm
steps=25
prices=average_cost_per_media
media = media_mix_model.media

media_maxes = media.max(axis=0) * (1 + percentage_add)
half_max_effective_concentration=media_mix_model._mcmc.get_samples()['half_max_effective_concentration'].mean(axis=0)
lag_weight=media_mix_model._mcmc.get_samples()['lag_weight'].mean(axis=0)
slope=media_mix_model._mcmc.get_samples()['slope'].mean(axis=0)
beta_media=media_mix_model.trace['beta_media'].mean(axis=0)

media_ranges = jnp.expand_dims(
      jnp.linspace(start=0, stop=media_maxes, num=steps), axis=0)

media_ranges=jnp.repeat(media_ranges,len(beta_media)).reshape(steps,len(beta_media),media_mix_model.n_media_channels, media_mix_model.n_geos)

media_response=beta_media*media_transform_hill_adstock(media_mix_model,
                                  media_ranges,
                           lag_weight,
                           half_max_effective_concentration=half_max_effective_concentration,
                           slope=slope, normalise=True)

You'll then have to plot an XY scatter plot where Y=media response and X = media spend (which is media*prices)

I hope this helps you all

[MuhammedTech]

Hi, thank you very much

On Wed, Jun 14, 2023 at 12:06 xijianlim @.***> wrote:

Hi, sorry for the delay in replying. Unfortunately I can't share the exact code because it is now my company's PI but i'll demonstrate how you can plot this.

— Reply to this email directly, view it on GitHub https://github.com/google/lightweight_mmm/issues/51#issuecomment-1590601232, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOSU3722LT3LAFPZLLI4HQLXLFPJFANCNFSM55JCH5DQ . You are receiving this because you commented.Message ID: @.***>

[xijianlim]

@pabloduque0 @steven-struglia @MuhammedTech and @hawkinsp I've created a collab book showing examples of how you can build functions to export these curves as well as channel tuning

https://colab.research.google.com/drive/1zKHmT_CR6AmVbH-4PdsmUJdumMadcmrO#scrollTo=WcYeesq4w3jT