Parameter with constraint and Parameter with dynamic max value?

[d4k0]

Hi,

I'm currently trying to solve 2 problems I have:

1.) One of the programs I'm trying to tune has the contraint that the dimension must be divisible by the block size (dimension % block_size == 0). I searched the issues and found https://github.com/jansel/opentuner/issues/106 where one of the proposed solutions for generating a parameter with constraints is to create a new parameter type which would be perfect.

However, I'm not quite sure where to implement the constraint. The PowerOfTwoParameter just checks in the constructor that the min and max values are a power of 2. After some looking through the code I found the set_value() function in NumericParameter:

def set_value(self, config, value):
  assert value >= self.min_value
  assert value <= self.max_value
  self._set(config, value)

Is this the function where OpenTuner checks if the generated parameter is valid? If yes, can I simply overwrite the function and add my constraint (max_value must be divisible by the generated value) there?

---

2.) Another program uses a block size parameter, but the max value depends on another parameter (to be exact the used algorithm; the list of algorithms is modelled as an EnumParameter).

There are only two possibilities for the interval:

1. [1,N]

2. [1,N - x]

N and x calculated at the beginning of the tuner (and don't change).

What would be the best way to model this? Should I simply create two parameters for the block sizes? Would OpenTuner be able to work properly with this? N is 245000 for the example I'm using, x would be 5600.

Or would it be better to use one parameter and somehow clamp/adjust the generated value when the interval [1,N - x] is needed? Or is there any other solution?

I would appreciate any help on my 2 problems.

[jbosboom]

I highly recommend trying approach 1 or 3 from jansel's comment in #106 first. Approach 3 is the clean way to handle this; approach 1 is less "nice" but seems to work fine in practice for many problems.

But assume you create a new parameter type and override set_value. What do you do when the technique tries to set a value for the parameter such that the constraint doesn't hold? (Leave aside for the moment how you'd read the value of the other parameter.) You can't raise an exception, because nothing is prepared to handle it; even if the technique had a handler installed, it doesn't know about your constraint, and so has no idea how to retry. You could round to the nearest legal value, but if you were okay with that, you'd just use approach 1. I don't see anything else you can do.

PowerOfTwoParameter works because it appears to the techniques as the base-2 log of its actual value (via scale/unscale, see ScaledNumericParameter), so there's no need to enforce a constraint; unscale takes care of the rounding.

---

For your second problem, either approach should work, assuming that increasing the value of the parameter means approximately the same thing in both cases. If N is much greater than x (as you say), then I'd try one parameter and clamp it.

If, on the other hand, algorithm A frobs more widgets when the parameter is large, while algorithm B frobs more when the parameter is small, use two parameters (or invert the sense of one of them, and use one). Techniques understand that some parameters may not have any effect. (Indeed, that's the normal case: even when all the parameters are used during fitness evaluation, sometimes some of them just don't matter.)

[d4k0]

Many thanks for your fast answer.

Approach 3 was also my preferred one, but I wasn't sure how to express the term B = p1 * p2 using IntegerParameter so that all generated values fulfill the equation.

Let's suppose the dimension is 10 (this is defined at the start of the tuner and doesn't change), so the possible block sizes are 1,2,5 and 10. How would you model this using two IntegerParameter with this approach? I can't imagine it right now as OpenTuner normally chooses values between the declared min and max values of the parameter. In jansel's equations the example would look like this (if I'm not mistaken):

A = p1 (= block size) B = p1 * 10 (= block size * fixed dimension)

In a later answer jansel mentions param.legal_range(config) which should be overridden for supporting dynamic ranges of parameters, but I'm still not sure how to really do this.

Approach 1 would also be an option, but I also couldn't figure out how this works as the normalize() simply seems to call itself (in ScheduleParameter it sets something).

---

The second program is an ODE solver. I believe the parameter (= block size) has the same effect for each algorithm, so this could probably work with one parameter.

Is there any (mathematical) "rule" to follow for clamping? Let's say one interval is [1,120] and the other is [1,100]. OpenTuner now generates the value 112 for the second interval. Should I just clamp it to 100? Or is the "relative position" in the interval also important here (especially when generating a value that lies in both intervals)?

And after clamping, should the new value be written back to OpenTuner/the current config? If yes, is there a special function for this?

[jbosboom]

Okay, I thought both the dimension and block size were varying during a tuning run. If the dimension is fixed, then you know the allowed values for the block size. You essentially want an EnumParameter (choose from a fixed list) that's searched like a numeric parameter (so, e.g., hill climbers do something useful).

First, I'd try using an EnumParameter and seeing if the results are good enough. (Depends on how many of them you have, how many choices each one has and the distance between the divisors.)

If that's not good enough, I would try an IntegerParameter with range from 0 to len(divisors), and interpret that (in your run method) as an index into the divisors. That way a technique can increase or decrease the value numerically. The scale of the search may not match the scale of the divisors (when the divisors are irregularly spaced), but I don't expect that to affect much.

If that's not good enough, you can implement a new ScaledNumericParameter and round to the nearest legal value in _unscale. But if the smallest or largest divisors are close together (i.e., 1 and 2 if the range is 0-1000), techniques may have a hard time getting to the extreme values because they're so close.

This would be a candidate for a new type of parameter, as it doesn't precisely fit any of the existing ones, but I don't see how to do better than the scaled approach.

---

If you're using one parameter, then its range is the union of the ranges, so in your example, [1, 120]. If OpenTuner generates {'algorithm': 'b', 'value': 112}, I would just clamp that in the run method and leave OpenTuner's configuration alone. The idea behind sharing a parameter that if both algorithms prefer a high value on the same inputs, techniques can quickly learn 'high values are good' independent of which algorithm is better.

I guess you could also try using a FloatParameter in 0.0-1.0 if it's the scales, rather than absolute values, which are comparable between algorithms. I don't think that's the case for a block size, though.

---

In general, it's not necessary to have a perfect encoding of the search space. OpenTuner does stochastic search, not constraint solving. Your time is better spent experimenting with what is easy to express with the existing parameters. Then by inspecting the produced configurations, you can come back to any parameters OpenTuner can't find good values for.