Different results Robyn 3.6 and 3.7 (same data and hyperparameters))

[tgtod002]

Project Robyn

I re-ran the same set of data using Robyn 3.6 then re-ran with 3.7. I received very different results. Was there a major change in the logic that would cause this?

Thanks.

Describe issue

Describe your issue and provide warnings and error messages.

Provide reproducible example

Issues are often related to custom input data that is difficult to debug without. If necessary, please modify your data to mask real values and share a dataset that is able to reproduce the issue. Please also share your model configuration and exported JSON files if available.

Environment & Robyn version

Make sure you're using the latest Robyn version before you post an issue.

• Check and share Robyn version: packageVersion("Robyn")
• R version (Please, check and share: sessionInfo() or R.version$version.string)

[laresbernardo]

Hi @tgtod002 we've changed a lot of things around since version 3.6. Check our changelogs based on our releases here.

Basically, we re-coded everything from scratch to migrate from data.tables to R and tidy code format. Results should be reproducible using the same versions, not guaranteed when different versions.

Keep in mind there are several random associated processes that will give different results; that's why we enable the seed parameter to ensure the same randomness when running random-related flows and getting the same outputs when re-run with the same inputs.

Lastly, when you say "very different results", it's not necessarily bad: remember Robyn (nevergrad) will run different scenarios and optimize to minimize the errors, returning thousands of possible models, and providing the best most different models for the user to pick from. They will all be mathematically the best, but not all will make sense necessarily; that's where the user must intervene with his/her knowledge of the business and pick the best one to continue.

[tgtod002]

Thanks for letting me know. It makes sense.

Given that, does it make sense to re-run the model every time new versions of Robyn is released?

[gufengzhou]

hi @tgtod002 , by "same hyperparameters" do you mean the same set of ranges at the beginning? if yes, then Bernardo is totally right. Due to the several random processes, even with the same version, you will/should receive different results. Although when your models are all converging, then the candidate clusters from different runs shouldn't be too different. But they won't be identical.

Yes, it's good exercise to rerun the model after update, because Robyn is still being actively developed. We'll do everything we can to ensure consistency

[gufengzhou]

Please reopen if this issue reoccurs.