data simulated vs holdout purchases

[AlexanderTrg]

Help to understand why data simulation on trained parameters shows a match. At the same time, the validation model is linearly larger than the hold-out?

1. why does repeated purchases (zero) on validation show a greater value than the hold-out, the simulation is the opposite?
2. what could be the reasons? any ideas?
3. could it be the reason that the behavior of buyers has changed a lot on the holdout?

p.s. the first drawing - the model was trained on the entire dataframe, the second drawing - the model was trained on calibration_period. Holdout - the next six months.

[ColtAllen]

Hey @AlexanderTrg ,

Development on lifetimes ceased 3 years ago. Here's the link to the successor library: https://github.com/pymc-labs/pymc-marketing

Have you looked at a time series plot of the simulated data prior to running it through the calibration_and_holdout_data function? My experience with the data simulation functions in lifetimes is that all customers are 'born' at the same time period T=0, which isn't particularly realistic.

I have a notebook for a marketing case study involving lifetimes, and I had to get creative with generating believable synthetic data. I hope you find this code helpful: https://github.com/ColtAllen/marketing-case-study/blob/main/case-study.ipynb

[AlexanderTrg]

thnx @ColtAllen checked out pymc-marketing lib. To be honest, I can't find if this is useful for me and close to my question) moreover, that lab uses the same approach (BG/NBD) So if U know or guess "why does repeated purchases (zero) on validation show a greater value than the hold-out, the simulation is the opposite?" could U clirify this?

[ColtAllen]

@AlexanderTrg did you take a look at the notebook I posted the link to? The Generate Synthetic Data section in particular uses lifetimes functions and is more relevant to this issue.

I only mentioned pymc-marketing to bring awareness to the fact it is Lifetime's successor going forward, but will admit it is still in early beta at this time. You are correct in that it contain its own BG/NBD implementation, in which we've fixed the known bugs in the lifetimes approach as explained here: https://github.com/CamDavidsonPilon/lifetimes/issues/442

[AlexanderTrg]

@ColtAllen, Hi, thanks for the help. Do I understand correctly that calibration_and_holdout_data is not working correctly, while plot_period_transactions is working well? The fact is that the simulation does not include the new users, as well as the churn of current users? The notebook that you threw off shows the impact of the marketing campaign, but it was not in my data. At the same time, with any time shift, the problem will persist - the model predicts larger sales than they actually occur (in comparison with the holdout). At the same time, plot_period_transactions - shows a perfect match ModifiedBetaGeoFitter helped a little, but did not solve the problem.

[AlexanderTrg]

update. I meant conditional_expected_number_of_purchases_up_to_time when has speking "calibration_and_holdout_data is not working correctly"

[ColtAllen]

@AlexanderTrg I think I understand what you're asking now. When I read your original message I assumed you were using beta_geometric_nbd_model_transactional_data() to generate simulated data from a trained model; sorry for the misunderstanding.

calibration_and_holdout_data() only has issues with monetary values. plot_period_transactions() works as intended, but it only compares model predictions to training/calibration data, not test/holdout.

Try running plot_cumulative_transactions() or plot_incremental_transactions() (the latter uses the raw transactional data instead of the summarized model input data). Both graphs will show how your dataset is changing over time, with a divider line for the calibration/holdout split. There are examples of both in the notebook I shared.

If there is a considerable difference between the two datasets, you may have a trend changepoint or seasonality in your data. We are working on ways to incorporate seasonality into pymc-marketing, but there's no way to control for it in lifetimes.

[AlexanderTrg]

@ColtAllen hello and thnx for your helping could you clarify why are plots so different? and in the 1 case model predict understates. In the other one overstates. why?

[ColtAllen]

The top plots applies the expected number of purchases as new customers are added, whereas the bottom plot compares the same customer's behavior in the training & test sets. Those kinks in the first plot seem to be happening at 365-day intervals, suggesting strong seasonal purchasing patterns are skewing your model.

Seasonality is in the roadmap for the pymc-marketing successor library, but lifetimes is no longer being actively developed. For now I'd suggest doing some EDA to find a way to control for seasonality.