Multiple dual optimizers

[juan43ramirez]

Enhancement

Enable the user to provide several partially instantiated "dual optimizers". These could be handled "as a single" optimizer by applying all operations currently applied to constrained_optimizer.dual_optimizer simultaneously across the set of optimizers.

See #45 for a PR which enables support for multiple primal optimizers.

Note that dual optimizers are currently instantiated internally by Cooper after the instantiation of dual variables by the formulation. For a LagrangianFormulation, this happens after a first call to cmp.closure.

Therefore, too much granularity or flexibility for the the grouping of dual variables under a shared optimizer is trickier than in #45. If restricted to groupings of constraints which are easily derived from the formulation/cmp (like inequality and equality constraints), this feature should be relatively easy.

Motivation

Setting up different optimizer classes or hyperparameters across (groups of) constraints. For instance, one for equality constraints and another for inequality constraints.

Original Idea

A natural follow-up feature for [#45] would be [to allow for] multiple (partially instantiated) dual optimizers.

It might be tricky to allow for too much granularity (say a different optimizer for each constraint). But allowing for different optimizers for equality and inequality constraints could be a good first step.

However, this does not seem to be a pressing feature to implement at the moment.

Originally posted by @gallego-posada in https://github.com/cooper-org/cooper/issues/45#issuecomment-1224909954