`BudgetOptimizer` equal budget across all channels

[AlfredoJF]

Just following up on the issue discussed below. I encountered the problem again using the new optimize_budget method. https://github.com/pymc-labs/pymc-marketing/blob/ccbd5a3ce454f370593804dfb061d6f1694db060/pymc_marketing/mmm/mmm.py#L2327

I have tried all possible ftol values (from 1e-9 to 1e9), but the budget remains equally allocated across all channels.

Originally posted by @emil-matti117 in https://github.com/pymc-labs/pymc-marketing/discussions/1234

[williambdean]

Interested in making a PR? @AlfredoJF

[cetagostini]

@AlfredoJF Do you have any information from the result object? I would be interested in analyse the gradient. On the other hand, how different are the curves for your channels in the model? It could be you are running out of iterations in a very difficult problem? 👀

[AlfredoJF]

@cetagostini I can share more details later, but for now, the minimize function stops after just one iteration. I deleted the PyTensor compile files for every ftol value I tested.

Any idea what might be going on or how to fix it?

[AlfredoJF]

@cetagostini here is the result object using the default minimize_kwargs

{
"x": [16239.31623932, 16239.31623932, 16239.31623932, 16239.31623932, 16239.31623932, 16239.31623932, 16239.31623932, 16239.31623932, 16239.31623932, 16239.31623932, 16239.31623932, 16239.31623932, 16239.31623932], 
"fun": -2.2250162881378923, 
"jac": [-7.90455746e-06, -4.96492540e-06, -6.82200684e-06, -5.51743075e-06, -2.08370294e-06, -2.99801214e-06, -6.26861003e-06, -1.61194180e-05, -5.97028182e-06, -8.26343134e-06, -3.41375200e-06, -5.71414876e-06, -6.25486236e-06], 
"nit": 1, 
"nfev": 1, 
"njev": 1, 
"status": 0, 
"message": "Optimization terminated successfully", 
"success": True
}

[cetagostini]

Yeah, based on the print looks like the Jac its very small. Given that the optimizer thinks its already in the local minimum. Can you try using a custom utility function which just compute the mean, you can use the utilities functions from PyMC Marketing and then add a random big value? Like in the following example:

def average_response(
    samples: pt.TensorVariable, budgets: pt.TensorVariable
) -> pt.TensorVariable:
    """Compute the average response of the posterior predictive distribution."""
    return pt.mean(_check_samples_dimensionality(samples)) * 1_000 # Random multiplier to increase response, avoid the derivative to be so small.


allocation, result = mmm.optimize_budget(
    budget=time_unit_budget,
    num_periods=num_periods,
    utility_function=average_response,
)

sample_allocation = mmm.sample_response_distribution(
    allocation_strategy=allocation,
    time_granularity=model_granularity,
    num_periods=num_periods,
    noise_level=0.01,
)

Whats the size of your budget respect to your response units? I feel this can be given if your budget is in millions and your response in hundreds. @AlfredoJF

[AlfredoJF]

Thanks @cetagostini what you said about a small jac totally makes sense.

Actually, I was already doing that but with the target scaler as you suggested in the MMM Hub. Then, I started to play around with the scaler, and it seemed like using at least * 10 the minimize function does not get stuck in the first iteration, but the optimisation takes longer (from 10s to 3-ish min) and also increased ftol=1e1.

Then, I tried with ftol=1e-10 without the custom scaler and produced the same result, also with a longer optimisation time.

[cetagostini]

Yeah, thats odd. I'll not expect this to came to often but really depends on how big are your units on spend and contribution. Regarding the optimisation time, does this happens changing the scaler from 10 to 1000? Like with 10 is 3-ish minutes and 1000 is 10s? if so, I actually would expect the opposite.

On the other hand, this trick only amplifies the information but should not change the result. So, play with it and verify using different scalers keeps the % distribution of your budgets the same. @AlfredoJF

[kb-open]

Thanks @cetagostini what you said about a small jac totally makes sense.

Actually, I was already doing that but with the target scaler as you suggested in the MMM Hub. Then, I started to play around with the scaler, and it seemed like using at least * 10 the minimize function does not get stuck in the first iteration, but the optimisation takes longer (from 10s to 3-ish min) and also increased ftol=1e1.

Then, I tried with ftol=1e-10 without the custom scaler and produced the same result, also with a longer optimisation time.

@AlfredoJF Are you saying that you just made one change (i.e., ftol=1e-10) and you could circumvent the issue? That is, you didn't need to add the custom utility function? And also, where did you change the ftol parameter? Could you please kindly point me to?

[cetagostini]

@kb-open ftol is part of minimize kwargs read here. You can access them using the parameter minimize_kwargs in the allocate function. You can see the code here!

Not sure about @AlfredoJF experience, but this should help because control the precision of the optimizer.