Gradient-Free-Optimizers
Gradient-Free-Optimizers copied to clipboard
SimonBlanke
Bug in Powell's Method: Limited angle when changing direction
In its current implementation of Powell's Method the change in direction does only happen orthogonally (or 90°) to the previous direction. To work properly it should allow a continuous range of angles.
Hi @SimonBlanke. Looking at this issue it seems that the search is restricted to one coordinate dimension at a time. Therefore when the algorithm switches dimensions via new_dim, the new search direction (along the next axis) is orthogonal (at 90 degrees) to the previous search direction (the previous axis).
Kindly let me know your view about this.
You can also assign this issue to me.
My proposed solution; Integrate a continuous line search at the end of each Powell cycle. Therefore at the end of each full cycle, we compute the net displacement.
@smilingprogrammer Could you elaborate on this? How would this line search look like?
My original goal of this issue was to implement the calculation of the angle close to how it is described in mathematical literature.
What i'm actually describing is also how i will be implementing it as described in mathematical literature.
As for the line search, i meant the “continuous‐angle” line search as described in Powell’s.
Hi @SimonBlanke , while trying to implement this issue, Must it be constrained to the current method we are using, or we can bring in new methods. Reason: The HillClimbOptmizer as used in the Powells Method seems to offer no deterministic guarantee of converging to the true minimum in my new implementation. So i am thinking if i should try out new ways
Update i have got it to work with the current HilClimbOptimizer. You can find the implementation here
From what i have read so far "golden section technique" is mostly used and it is guaranteed to converge to the minimum within a given interval. But it is slow in complex functions.
The HillClimbOptmizer as used in the Powells Method seems to offer no deterministic guarantee of converging to the true minimum in my new implementation.
Good observation! The neighbour selection for hill-climbing based algorithms in GFO is stochastic. This should not hold you back implementing the new angle calculation.
You can find the implementation
I am looking forward this. Could you open a PR for your changes?