Writing the mathematical formulation is a nightmare

[manuelma]

I would like to consider an alternative approach where we'd write the mathematical formulation just for the case of a single temporal block with flat resolution and single stochastic structure with one scenario (and add a note at the beginning where we explain, in general terms, how multiple variable resolutions and complex stochastic structures would affect the formulation).

I believe that would be much easier to read too, but especially, it would dramatically simplify the life of the developer who already has to write complex code and tests, and on top of that, at the moment is also expected to update the latex with the math.

[jkiviluo]

Yes, I fully agree. The temporal and stochastics can be just expressed with one index in stylized equations. Naturally separate part that explains what lies behind that index.

[manuelma]

Anybody interested ? @tarskul @datejada @gnawin @g-moralesespana ?

The work would be to simply rewrite the entire mathematical formulation in the simpler terms described above (single scenario, flat resolution). I can write the section about how the complex structures would affect the constraint and variable indexing...

[tarskul]

I think that something like that is indeed best left for us given that you get the most work done in the code.

I was definitely planning to look at the entire formulation as part of the total revision of the documentation. (Although I must note that so far I've had troubles getting it in my actual schedule.)

But I'm also a bit confused as to what the problem is. Typically I write something like this:

for all n, u, s, t:
    v_n,u,s,t > 0
with:
    n: node
    u: unit
    s: scenario
    t: time

So, why would the temporal and stochastic indices cause troubles? Do you mean that the constraint are actually conditional with multiple if-else statements? i.e. v_n,u,s,0 = 0

I'm not sure how to feel about the simplification. From an academic point of view the theoretical behaviour is typically more important than the actual implementation... But I suppose it can be ok because it is not wrong, just simplified...

[manuelma]

Thank you @tarskul !

The problem is the size of the formulas. My idea is if we just assumed flat resolution and single scenario we'd be able to reduce them enormously. Then the developer would spent a fraction of the time they spend now in writing/updating the mathematical formulation. The user would also be happier since they'd be able to read the formulas and get the idea of the constraint, without worrying about the details of temporal (dis)aggregation and stochastic path unpacking in every term of each constraint, as they have to do now.

Maybe I can write one or two math formulations the way I propose so you can see the benefit, and if everybody is onboard then you guys can write the rest?

Edit: And to keep things correct, we'd add a note to every constraint stating our assumptions, together with a link to a section describing temporal (dis)aggregation and stochastic path indexing in general.

[g-moralesespana]

I also fully agree. It took me a lot of time to try to understand the documentation when I first checked it. That is not a good signal. It should be way simpler, so you do not need to be a math expert + very familiar with SpineOpt already + even know Julia to be able to understand the documentation.

[tarskul]

@manuelma an example would indeed be helpful and perhaps make it sound less scary. And I admit that it would indeed be nice if we can make it easier to understand.

[datejada]

@tarskul I can help with that part. A "vanilla version" of the mathematical formulation is a better approach for new users. Then, the users have the sections for Temporal and Stochastic structures for conceptual details and the code for implementation details. We can talk in the next week's meeting to organize ourselves on how to proceed.

[manuelma]

I have updated the math for a few constraints already:

• unit_flow_capacity - https://spine-tools.github.io/SpineOpt.jl/latest/mathematical_formulation/constraints_automatically_generated/#constraint_unit_flow_capacity

• minimum_operating_point -https://spine-tools.github.io/SpineOpt.jl/latest/mathematical_formulation/constraints_automatically_generated/#constraint_minimum_operating_point

• ramp_up -https://spine-tools.github.io/SpineOpt.jl/latest/mathematical_formulation/constraints_automatically_generated/#constraint_ramp_up

• ramp_down -https://spine-tools.github.io/SpineOpt.jl/latest/mathematical_formulation/constraints_automatically_generated/#constraint_ramp_down

Please let me know what you think.

[datejada]

@manuelma it looks much readable, thanks! This will help future users.

I suggest a slight change if you agree it is functional. We can be more verbose in the conditions of the sums to make it easier to follow. I show an example using the unit flow capacity constraint in the following equations.

Also the minimum_operating_point uses $!IRN$ and it should be $\neg IRN$

\begin{aligned}
& \sum_{
        n \in ng : not\_reserve(n)
} {unit\_flow}_{(u,n,d,s,t)} + \sum_{
        n \in ng : nonspin\_upwards\_reserve(n)
} unit\_flow_{(u,n,d,s,t)} \\
& \le \\
& UC_{(u,ng,d,s,t)} \cdot UAF_{(u,s,t)} \cdot UCCF_{(u,ng,d,s,t)} \\
& \cdot ( \\
& \qquad units\_on_{(u,s,t)} \\
& \qquad + \left(1 - SDL_{(u,ng,d,s,t)}\right) \\
& \qquad \cdot \left( units\_shut\_down_{(u,s,t+1)}
+ \sum_{
    n \in ng: nonspin\_reserve(n)
} nonspin\_units\_shut\_down_{(u,n,s,t)} \right) \\
& \qquad - \left(1 - SUL_{(u,ng,d,s,t)}\right) \cdot units\_started\_up_{(u,s,t)} \\
& ) \\
& \forall (u,ng,d) \in indices(UC)
\end{aligned}

where

• not_reserve = $\neg IRN_{(n)}$
• nonspin_upwards_reserve = $IRN_{(n)} \land UR_{(n)} \land \neg INS_{(n)}$
• nonspin_reserve = $IRN_{(n)} \land INS_{(n)}$
• IRN = is_reserve_node
• UR = upward_reserve
• INS = is_non_spinning
• UC = unit_capacity
• UAF = unit_availability_factor
• UCCF = unit_conv_cap_to_flow
• SUL = start_up_limit
• SDL = shut_down_limit

[tarskul]

I've also looked at it, but I want to look at it a bit more. In the meanwhile I can say that I am thinking in the same direction as @datejada I would also replace the 'v_' with the name of the variable, the 'p_' with capital letters and move the indices to the subscripts.

One of the things I struggle with is the verticality of the terms. i do get that it is because of how the documentation is built but it is not ideal. Maybe it helps to put the operators in a different align part upfront (perhaps only the most important one, i.e. the inequality)? I'll play around with a few variations.

Also I typically put the 'for all' part upfront. Is that just me? Instead of indices(UC) I would define a new group and then also explain that group in the 'where' part.

[jkiviluo]

My preference is to write equations like this:

Using capitals to indicate parameters is not as readable as distinguishing with v and p. There are issues with the way the indexes get rendered, but if it could be made to look like this example of mine, I think it would be great. It's not much longer than using abbreviations and I really feel abbreviations make it lot harder to read quickly (since you often have to back and forth with the nomenclature).

[manuelma]

@tarskul have you looked at the latest version? some of your suggestions are already there.

[manuelma]

That's a good idea @jkiviluo - but I'm afraid our parameter names are so long that it will get ugly really quickly.

The overleaf about ramps and reserves used all caps abbreviations and I thought they were nice. Since the nomenclature is right there below the constraint, I think it's not a big inconvenience - it can be read quickly - plus you get a list of all the parameters involved in the specific constraint for free, which I think it is also useful information.

I will see how it looks for a few constraints with your proposal though, maybe I like it.

[tarskul]

@manuelma Ah, indeed it has changed since the last time I checked. It seems that I can't keep up with you. `:-)

@jkiviluo That is also an option we can consider. edit: @manuelma was also here quicker to comment than I so I don't need to finish my thought `:-)

[jkiviluo]

You wouldn't need to use the full long names - they could be a reasonable shorthand. The nomenclature could still be there to show the long form name.

I think it's really lot nicer to read when you don't need to go back and forth with nomenclature. With long equations you can also start to forget the earlier abbreviations and it's just a mess. I think that's one of the reasons I rather read equations from code than from formulations.

[manuelma]

I'm trying your suggestion @jkiviluo - seems like the way to go...

[jkiviluo]

If you can get the formatting to have the superscript and subscripts aligned on top of each other, that would be great!

[manuelma]

Maybe we can use the Iverson bracket to lift the conditions from the sumation subscript? They are really hard to read. Defining a new set is an option as @datejada is suggesting too.

This is how it looks with the Iverson bracket, what do you think? I kinda like it:

https://spine-tools.github.io/SpineOpt.jl/latest/mathematical_formulation/constraints_automatically_generated/#constraint_unit_flow_capacity

[gnawin]

Maybe we can use the Iverson bracket to lift the conditions from the sumation subscript? They are really hard to read. Defining a new set is an option as @datejada is suggesting too.

This is how it looks with the Inverson bracket, what do you think? I kinda like it:

https://spine-tools.github.io/SpineOpt.jl/latest/mathematical_formulation/constraints_automatically_generated/#constraint_unit_flow_capacity

Just jumping in to say that I like this new form, which looks a lot cleaner.

[jkiviluo]

Great!

[tarskul]

Maybe we can use the Iverson bracket to lift the conditions from the summation subscript?

Instead of inversion brackets, could we make it a multiplication with binary parameters? (similar to binary variables?)

[datejada]

Mostly done in commit https://github.com/spine-tools/SpineOpt.jl/commit/2909c6bad6a0369b89d1c96edf115e098f761382