offgridders Feature critical demand

Adresses issue #162

The following subtasks have been implemented:

[x] Defined title_demand_ac_critical and title_demand_dc_critical in the excel
[x] Define constants in https://github.com/rl-institut/offgridders/blob/a8fa6f1a6b8319962e65505320f9b47267fbef0a/src/constants.py#L45
[x] Added columns for demand_ac_critical and demand_dc_critical in input timeseries file
[x] Run simulation, check that new parameters are in simulation_experiments.csv (output)
[ ] If not, needs to be implemented in D0 https://github.com/rl-institut/offgridders/blob/dev/src/D0_process_input.py
[x] Check that timeseries was parsed correctly, add DEMAND_PROFILE_AC_CRITICAL and DEMAND_PROFILE_DC_CRITICAL
[x] Add generate_demand for new critical demands, see https://github.com/rl-institut/offgridders/blob/a8fa6f1a6b8319962e65505320f9b47267fbef0a/src/G1_oemof_create_model.py#L96 --- For later ---
[ ] Run a simulation with demands split into normal and critical demand to compare if KPI like demand_total stay the same (do not use shortage here), and also which do not. We might have to adapt KPI definitions accordingly (G3, G3a, G3b`)
[ ] Add a constraint that allows shortage to replace non-critical demands (G2b)
[ ] Think of KPI that represents how much non-critical demand is not covered. Should this be in total_shortage etc?
[ ] Add plausability tests, ie. is critical demand always supplied, and is the shortage activated ADDITIONALLY? (G3a)
[ ] Define benchmark test, ie. scenarios where you already can estimate some results and with which we can test that the new feature works.
[ ] Make sure that all excel files in the repo now are up-to-date and have the new parameters
[ ] Make sure that it is possible to parse excel files that do not have the new parameters. Give a warning about outdated input files.
[ ] Update the changelog.md
[ ] Use black for linting

Nov 30 '21 11:11 adnanalakori

Hi @adnanalakori! I edited the first post according to what is common use in github. There is a specific way to reference the issues that you are working on with a hashtag, and the to-do-list are clickable.

It may be that you do not want to adress each of tasks in this PR. Then delete some of the rows again. The PR can be closed if you can make sure that your changes do not interfere with the existing functionalities.

Dec 01 '21 08:12 smartie2076

Hi @smartie2076 , thanks! I unified names demand_ac_non_critical etc in the inputs/timeseries/test_site.csv, test_sitie.csv, and also constants in https://github.com/rl-institut/offgridders/blob/a8fa6f1a6b8319962e65505320f9b47267fbef0a/src/constants.py#L45 The simulation up to now runs well and checked in simulation_experiments.csv.

Dec 01 '21 10:12 adnanalakori

Can you upload the simulation_experiments.csv?

I wonder if it did load the timeseries of demand_ac_critical and demand_dc_critical (ie. a vector of 8760 entries), or only the title_demand. We need to make sure that it writes the vector to the data, so that you can process it later.

I think we might need to add it to https://github.com/rl-institut/offgridders/blob/dev/src/B_read_from_files.py#L551 in B, or also some processing in D0 to really get it into simulation_experiments.

(I am not sure if the following two points are really checked, or if some more work has to be done)

[x] If not, needs to be implemented in D0 https://github.com/rl-institut/offgridders/blob/dev/src/D0_process_input.py

[x] Check that timeseries was parsed correctly, add DEMAND_PROFILE_AC_CRITICAL and DEMAND_PROFILE_DC_CRITICAL If you want an easier time to check whether or not the vectors/timeseries for the different demand categories are parsed, you could also have different timeseries for each of them (for now you just copied the columns in the timeseries input file).

Dec 01 '21 13:12 smartie2076

Can you upload the simulation_experiments.csv?

@smartie2076 , please find a cope of the simulation_experiments.csv attached. It seems up to now the title is shown:

Regarding the next point, I did not consider it as the new parameters have been shown in simulation_experiments.csv

If not, needs to be implemented in D0 https://github.com/rl-institut/offgridders/blob/dev/src/D0_process_input.py

Also same apply to this point -> I understood "parsed correctly" that the simulation run with no errors:_

[x] Check that timeseries was parsed correctly, add DEMAND_PROFILE_AC_CRITICAL and DEMAND_PROFILE_DC_CRITICAL

Dec 01 '21 13:12 adnanalakori

You do not have to quote reply all of it, just the parts you want to hightlight ;)

Ah, I realized that actually we can not figure out if it worked with sensitivity_experiments alone. :( Can you upload the log-file? Directly with drag-and-drop, then I can also read it fully.

I am sure, actually, that we have to adapt this function: `https://github.com/rl-institut/offgridders/blob/feature-critical-demand/src/D0_process_input.py#L234 ...but, sadly, I was not very good at programming at this point, and there are very few logging messages, and the code is repetative. There are two options:

Just adapt everything that exists there for the initial title_demand_ac and title_demand_dc
Do it properly and generalize functions, to have less lines of code

There should definitly be more logging messages. So, try to understand the idea of the function and add some logging messages that help you understand which dara is added to the input data here. To observe what is changed in the function, you can add

import pp
pp.print(experiments_s)

Right below https://github.com/rl-institut/offgridders/blob/4d21c9e35f94ed4847d4492d769d1e03efbfe567/Offgridders.py#L117, and then you can check if you added the timeseries properly.

You can also check if you can store the experiments_s as a json file, which would be easier to read:

    json_data = json.dumps(
        experiments_s,
        skipkeys=False,
        sort_keys=True,
        indent=4,
    )

Dec 01 '21 16:12 smartie2076

If you choose to improve the offgridders code and go for...

* Do it properly and generalize functions, to have less lines of code

You should push the related commits in a seperate PR, ideally WITHOUT the demand_ac_cricical, which we would then merge into dev. Then we pull the new, nice dev with improved functions and logging messages into this branch, and continue working. That way, this PR does not get heavy with "maintenance work" within the Offgridders code.

Dec 01 '21 17:12 smartie2076

Hi @smartie2076 , It seems the error below because when we merged 4 similar functions in one and added demand_type in DO line 361, so we might need to mention it in line 476, would that help? Still not clear how it should be recognized as a demand for AC and DC and the demand_type have not been defined in constant.py?

def update_demand_profile(experiment_s, experiment, demand_profile, demand_type): experiment_s[experiment].update( { demand_profile: pd.Series( experiment_s[experiment][demand_type][0 : len(index)].values, index=index, ) return

Mar 29 '22 09:03 adnanalakori

Hi @adnanalakori! Thanks for posting the issue here - it makes it easier to follow up on. Ideally, when posting issues here, you add links to the appropriate code passages, so that it is absolutely without possibility of error clear which lines of code you are referring to. For that, you open the file in github (here D0), go to the appropriate line, click on the line number, click on the three points and click "copy permalink":

grafik

I add the links below, so that we are for sure talking about the same thing.

merged 4 similar functions in one and added demand_type in DO line 361

We are talking about this function:

https://github.com/rl-institut/offgridders/blob/ab336238af380d9755468f6da9205e809ef7a9c5/src/D0_process_input.py#L361

Thing is, it is not a proper function definition - you can (almost) only define functions on the first level (no tabs in front of def), but here you define it within a if-elif-loop:

elif experiment_s[experiment][FILE_INDEX] == None:
            # limit based on index

            def update_demand_profile(experiment_s, experiment, demand_profile, demand_type):
                experiment_s[experiment].update(
                    {
                        demand_profile: pd.Series(
                            experiment_s[experiment][demand_type][0: len(index)].values,
                            index=index,
                        )
                    }
                )
                return

So, move this function out of your if-else-loop, and only call the function within this if-else-loop. Otherwise, the function is not recognized.

so we might need to mention it in line 476, would that help?

Line 476 would be this, so I guess the line where you got an error.

https://github.com/rl-institut/offgridders/blob/ab336238af380d9755468f6da9205e809ef7a9c5/src/D0_process_input.py#L476

Still not clear how it should be recognized as a demand for AC and DC and the demand_type have not been defined in constant.py?

I think the problem is that you removed 4 similar code snippets and replaceAd them by D0.update_demand_profile, but then did not add the new calls of exactly this function - therefore, none of DEMAND_PROFILE_AC, DEMAND_PROFILE_DC, DEMAND_PROFILE_AC_CRITICAL, DEMAND_PROFILE_DC_CRITICAL are added to the experiment. Add the function calls, maybe this fixes the error.

grafik

Mar 29 '22 11:03 smartie2076

[ ] Add a constraint that allows shortage to replace non-critical demands (G2b)

Means adding constraint that allows capacity dispatch to supply critical load first and then non-critical load after that? The constraints shall work as a check point, correct?

Mar 31 '22 08:03 adnanalakori

[ ] Think of KPI that represents how much non-critical demand is not covered. Should this be in total_shortage etc?

I assumed that the amount of “non-critical demand is not covered” is already fixed at input constant as scenarios e.g. 10% 30% etc.? Do we still need to apply a KPI here?

Mar 31 '22 08:03 adnanalakori

[ ] Add plausability tests, ie. is critical demand always supplied, and is the shortage activated ADDITIONALLY? (G3a)

I assumed that the critical demand always supplied. Do we still to implement something here as well? could you elaborate on?

Mar 31 '22 08:03 adnanalakori

Regarding model inputs and outputs, I could imagine the followings:

Model inputs:

We are willing to implement various shortages in the input constant excel file shortage_max_allowed (10%, 30%, etc.). However this has to be applied only for non-critical load (we named it in our coding as DEMAND_AC and DEMAND_DC), is it ok?

Model outputs:

I assume that we are able to visualized electricity flow, etc. for both normal demand and critical demand as well, correct?
Techo-economic results will be the same as the current version of Offgridders?

Mar 31 '22 08:03 adnanalakori

[ ] Add a constraint that allows shortage to replace non-critical demands (G2b)

Means adding constraint that allows capacity dispatch to supply critical load first and then non-critical load after that? The constraints shall work as a check point, correct?

Means that the shortage that is allowed can only be as high as non-critical load only, so that critical load is always supplied. The constraints are defined in G2b, but I would have to check more in detail what needs to be changed. It can be that the shortage is included implicitly within the stability constraints. In that case, all stability constraints need to be changed.

The constraint https://github.com/rl-institut/offgridders/blob/76173e0fb081960a6f232f9c4bdc3c2f8698cb2d/src/G2b_constraints_custom.py#L959 Is not implemented or used, so do not just change that, it will not help. However, it also seems to me that the "timestep" constraint is anyway added with https://github.com/rl-institut/offgridders/blob/76173e0fb081960a6f232f9c4bdc3c2f8698cb2d/src/G2a_oemof_busses_and_componets.py#L113? Because the nominal capacity, not the sum over the whole year is limited my the "max_shortage". I would have to put some time into this to know for sure. It would surprise me a bit, because if I have a household with blackouts, the simulation should terminate if the shortage source was not able to cover demand, but this did not happen.

Sidenote, can also be moved into an own issue

I would greatly appreciate if this PR could do a fix and rename the variable SHORTAGE_LIMIT to STABILITY_LIMIT. Maybe start here and replace all occurrences in the code.

https://github.com/rl-institut/offgridders/blob/76173e0fb081960a6f232f9c4bdc3c2f8698cb2d/src/constants.py#L152

I am not sure how this came to be, but it is very, very misleading in G2b, for example: https://github.com/rl-institut/offgridders/blob/76173e0fb081960a6f232f9c4bdc3c2f8698cb2d/src/G2b_constraints_custom.py#L64

Mar 31 '22 08:03 smartie2076

[ ] Add plausability tests, ie. is critical demand always supplied, and is the shortage activated ADDITIONALLY? (G3a)

I assumed that the critical demand always supplied. Do we still to implement something here as well? could you elaborate on?

Yeah, if you implement is correctly it is. But there are some parameters/KPI, that should take this into account - for example the % of shortage should then not be calculated based on shortage/non-cricitcal demand but `shortage/(non-critical + critical demand).

Mar 31 '22 08:03 smartie2076

Regarding model inputs and outputs, I could imagine the followings:

Model inputs:

* We are willing to  implement various shortages in the input constant excel file `shortage_max_allowed `
  (10%, 30%, etc.). However this has to be applied only for non-critical load (we named it in our coding as DEMAND_AC and DEMAND_DC), is it ok?

That is the idea.

Model outputs:

* I assumed that we are able to visualized electricity flow, etc. for both normal demand and critical demand as well, correct?

You have to implement this into the plots, only if you do it will be plotted.

* Techo-economic results will be the same as the current version of Offgridders?

Some KPI calculations might have to be changed, as they have the demand as a denominator, which currently is only the non-critical demand.

Mar 31 '22 08:03 smartie2076

[ ] Add a constraint that allows shortage to replace non-critical demands (G2b)

Means adding constraint that allows capacity dispatch to supply critical load first and then non-critical load after that? The constraints shall work as a check point, correct?

This would be added in G2b and called under # ------------Stability constraint------------# in G1.build (around line 400)?

Apr 04 '22 07:04 Bachibouzouk

This would be added in G2b and called under # ------------Stability constraint------------# in G1.build (around line 400)?

Yeah it would. I am just not sure if it is basically already in the different stability constraints that I defined, or if it is not. Stability constraints would be: https://github.com/rl-institut/offgridders/blob/76173e0fb081960a6f232f9c4bdc3c2f8698cb2d/src/G2b_constraints_custom.py#L52 https://github.com/rl-institut/offgridders/blob/76173e0fb081960a6f232f9c4bdc3c2f8698cb2d/src/G2b_constraints_custom.py#L242 https://github.com/rl-institut/offgridders/blob/76173e0fb081960a6f232f9c4bdc3c2f8698cb2d/src/G2b_constraints_custom.py#L431

They are always written with a validity-check that has to pass after the optimization finished, eg: https://github.com/rl-institut/offgridders/blob/76173e0fb081960a6f232f9c4bdc3c2f8698cb2d/src/G2b_constraints_custom.py#L167

You would call the constraint in G1, if you did have to define a new constraint, I would probably add it at the end, so around here https://github.com/rl-institut/offgridders/blob/76173e0fb081960a6f232f9c4bdc3c2f8698cb2d/src/G1_oemof_create_model.py#L514

Apr 04 '22 09:04 smartie2076

I just noticed that you should make sure that your code still works with the old input template - ie. that if title critical demand (etc) is not included, that value is set to None. Same for the old title demand AC/DC.

Apr 26 '22 13:04 smartie2076