Reorganization

[michaelbynum]

What does everyone think about a slight reorganization of Egret? I am thinking of the directory structure below. I have not fully flushed it out, but I think you will get the idea. I don't think the current structure is intuitive, but the structure I outlined below might be missing some complexities.

• Egret
  • data
  • solvers
    • lazy ptdf
    • acopf
    • ...
  • models
    • acopf
      • declarations
      • models
      • useful acopf things
    • commitment
    • dcopf
    • bilevel
    • ...

[michaelbynum]

Also, I don't think such a reorganization would take too much time. It would primarily be moving files and changing imports. The only part that might be difficult is separating out the lazy ptdf solver. I have some thoughts on that topic. We could use custom Pyomo blocks to separate the model from the algorithm. The custom block would define an API that allowed the solvers to do what they needed.

[bknueven]

I'm certainly not opposed to reorganizing Egret -- I think the current organization has been at its limits for some time. I think we should also add some high-level imports for ease-of-use.

Would the "solvers" folder just be for meta-solvers? If declarations is moving inside a models sub-directory, does this mean splitting out things currently in model_library? What about variables constraints shared between models?

I would be curious what could be done with lazy ptdfs. Keep in mind the complexity with unit commitment -- each "time-step" is an independent problem, although in the current implementation they share data if their network topology is the same. I think to keep a reasonable memory footprint we would want to maintain this feature.

[michaelbynum]

I think we should also add some high-level imports for ease-of-use.

I completely agree.

Would the "solvers" folder just be for meta-solvers?

I think so?

If declarations is moving inside a models sub-directory, does this mean splitting out things currently in model_library? What about variables constraints shared between models?

Good question. I'm not sure how I feel about this yet. There is very limited sharing between acopf and dcopf (just some variables, really). These two libraries should be pretty well contained. Other modules, such as unit commitment, should be able to use them without any trouble. We could get into trouble if we end up with two module that need each other. Thoughts?

I would be curious what could be done with lazy ptdfs. Keep in mind the complexity with unit commitment -- each "time-step" is an independent problem, although in the current implementation they share data if their network topology is the same. I think to keep a reasonable memory footprint we would want to maintain this feature.

This will require a more in depth discussion, I think.

[bknueven]

There is very limited sharing between acopf and dcopf (just some variables, really). These two libraries should be pretty well contained. Other modules, such as unit commitment, should be able to use them without any trouble. We could get into trouble if we end up with two module that need each other. Thoughts?

If these two modules are reasonably self-contained, then I say we go for it. We can avoid recursive dependency between modules so long as there was a clear choice on where the current and any new shared functions go.

With regards to the lazy PTDFs, we could mimic what happens now -- each block would keep a reference to some PTDF object, which stores the data and has some useful methods for interacting with it. Blocks share PTDF objects, such that identical data is not duplicated in memory. When a new block is created, we look and see if there are any existing PTDF objects which match our transmission topology. This is essentially what happens now for the unit commitment model, which has time-indexed blocks for easy interfacing with the OPF constraint functions.

[jeanpaulwatson]

Agree with most of everything above. Would avoid the name "solver" because it implies something we don't ship / provide. Would prefer to have self-contained acopf and dcopf libraries, given the very limited redundancy involved. Definitely think the lazy PTDF discussion is much more involved than we can do here.