avalanche icon indicating copy to clipboard operation
avalanche copied to clipboard

Unbalanced Steps Sizes Option for Benchmarks Generators

Open vlomonaco opened this issue 4 years ago • 2 comments

I noticed it is quite tricky at the moment to generate a benchmark with an unbalanced number of examples for each step. It would be nice to have an option in ni_scenario, nc_scenario and similar to set the number of examples for each step.

vlomonaco avatar Feb 23 '21 17:02 vlomonaco

Now we have the data_incremental_benchmark utility that, starting from a benchmark instance, allows for the creation of a new benchmark instance in which experiences are split into mini-experiences of a given fixed size.

I think that making a more abstract version of that helper should work. We can reduce the size of the experiences (thus discarding instances) or we can allow the user to create an unbalanced splitting. @vlomonaco what was your original idea?

lrzpellegrini avatar Jun 03 '21 10:06 lrzpellegrini

Yes I think being able to create an unbalanced split would be ideal maybe just re-distributing examples of the same class within experiences that contain that class? It's difficult to came up with a general solution for this as different benchmarks may need a different "redistribution" policy.

vlomonaco avatar Jun 03 '21 10:06 vlomonaco