Presentation

The objective of this repository is to share with an MIT license the visualization tools used with public data and developed by the Wholesale Markets Surveillance Directorate (DSMG) of the Regulatory Commission of Energy (CRE). It can be used by final users such as developers and energy analysts.

To obtain the proposed interactive visualizations, the user should :

• install the package,
• download the input data,
• run one of the scripts with ipython and her own choice of parameters.

All suggestions are welcome at opensource[at]cre.fr.

Installation

It can be installed with :

cd ~/Downloads
git clone https://github.com/cre-dev/pub-data-visualization.git
cd pub-data-visualization
python3 -m venv venv
source venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install -e .

The installation can then be tested with one of the following :

python3 scripts/weather/main_curve.py

or

python3 scripts/load/main_forecasting_error.py

The scripts should terminate without any error. They will create 3 folders in the home directory :

~/
├── _energy_plots/       # for the plots
├── _energy_public_data/ # for the raw data
├── _energy_tmp_data/    # for the transformed data

However, ipython should then be preferred for interactive plots.

Download the input data

The data used for the visualizations proposed in this repository come from different public data sources. only the data from eCO2mix and Météo-France are downloaded automatically. Data from RTE and ENTSO-E should be downloaded manually.

eCO2mix

Data about the supply and demand equilibrium and provided by Réseau de Transport d’Electricité (RTE) through eCO2mix allow to illustrate the production and the consumption on the French electricity network. They can be downloaded automatically. No account is necessary.

ENTSO-E

The European Network of Transmission System Operators for Electricity (ENTSO-E) publishes fundamental data on its transparency platform. The source files used for the visualizations in this repository currently have to be downloaded manually with the SFTP share. An account is necessary.

Météo-France

As the French national meteorological service, Météo-France provides observation data extracted from the Global Telecommunication System (GTS) of the World Meteorological Organization (WMO). The data can be downloaded automatically. No account is necessary.

RTE

RTE publishes fundamental data about the French electricity transmission system. The files currently have to be downloaded manually on the platform RTE services portal. An account is necessary.

Local organization of the data

The data have to be stored as follows :

~/_energy_public_data/
├── 11_ENTSOE/
│  ├── ActualGenerationOutputPerGenerationUnit_16.1.A/
│  │  ├── 2021_01_ActualGenerationOutputPerGenerationUnit_16.1.A.csv
│  │  ├── …	 
│  ├── ActualTotalLoad_6.1.A/
│  │  ├── 2021_01_ActualTotalLoad_6.1.A.csv
│  │  ├── …	 
│  ├── DayAheadPrices_12.1.D/	 
│  │  ├── 2021_01_DayAheadPrices_12.1.D.csv
│  │  ├── …	 
│  ├── Outages/	 	 
│  │  ├── UnavailabilityOfGenerationUnits_15.1.A_B/	 
│  │  │  ├── 2021_01_UnavailabilityOfGenerationUnits_15.1.A_B.csv
│  │  │  ├── …	 
│  │  ├── UnavailabilityOfProductionUnits_15.1.C_D/	 
│  │  │  ├── 2021_01_UnavailabilityOfProductionUnits_15.1.C_D.csv
│  │  │  ├── …	 
├──20_MeteoFrance/	 	 
│  ├── synop/	 	 
│  │  ├── postesSynop.csv
│  │  ├── synop.201001.csv
│  │  ├── …	 
├──24_RTE/	 	 	 
│  ├── Centrales_production_reference/
│  │  ├── Centrales_production_reference.xls
│  ├── DonneesIndisponibilitesProduction/
│  │  ├── DonneesIndisponibilitesProduction_2010.xls
│  │  ├── …	 
│  ├── eCO2mix_RTE/	 
│  │  ├── eCO2mix_RTE_Annuel-Definitif_2012.xls
│  │  ├── …	 
│  ├── ProductionGroupe/	 
│  │  ├── ProductionGroupe_2012/
│  │  │  ├── ProductionGroupe_2012-semestre1.xls
│  │  │  ├── …
│  │  ├── …

How-to : ready-to-run examples

In this repository, we propose a set of modules that read, format, transform and plot the input data from different public sources. We also provide ready-to-run visualization scripts as illustrated below. The parameters therein can be modified by the user.

Indices

Day-ahead fixing prices

This figure, that represents the fixing prices of the day-ahead auctions, is obtained by running scripts/indices/main_price.py. The data, provided by ENTSO-E, currently have to be downloaded manually.

Load

Load curve

This figure, obtained by running scripts/load/main_power.py, is a mere representation of the load curve. Data from eCO2mix and ENTSO-E can serve as inputs.

Day-ahead forecasting error

This figure represents the national load forecasting error and is obtained by running scripts/load/main_forecasting_error.py. The data are provided by eCO2mix.

Outages

Animated view of the unavailability

This figure is obtained by running scripts/outages/main_animated_availability.py. The data have to be downloaded manually from ENTSO-E or RTE platforms.

Evolution of the mean unavailability

This figure is obtained by running scripts/outages/main_evolution_mean_availability.py. The data have to be downloaded manually from ENTSO-E or RTE platforms.

Expected availability program of a given unit

This figure is obtained by running scripts/outages/main_expected_program.py. The data have to be downloaded manually from ENTSO-E or RTE platforms.

Availability programs

This figure is obtained by running scripts/outages/main_incremental_programs.py. The data have to be downloaded manually from ENTSO-E or RTE platforms.

Regression delays

This figure is obtained by running scripts/outages/main_regression_delays.py. The data have to be downloaded manually from ENTSO-E or RTE platforms. It displays a linear fit between the initially announced and the finally observed length of the outages. The coefficient being obtained with the minimization of a squared error, outliers have a significant effect.

Production data

Unit production

This figure is obtained by running scripts/production/main_power.py. It can be used with data provided by eCO2mix, ENTSO-E or RTE.

Weather data

National mean weather

This figure is obtained by running scripts/weather/main_curve.py. The data, provided by Météo-France, are downloaded automatically.

Distribution of the temperature

This figure is obtained by running scripts/weather/main_distribution.py. The data, provided by Météo-France, are downloaded automatically.

Multiplots

Spot report

This figure is obtained by running scripts/multiplots/main_spot_report.py. As it mixes data from different sources, the data from ENTSO-E and RTE have to be downloaded manually.

Announced availability and observed production of a given unit

This figure is obtained by running scripts/multiplots/main_transparent_production.py. The data from RTE have to be downloaded manually.

2D-distribution of price and load

This figure is obtained by running scripts/multiplots/main_scatter_price_load.py. The data from ENTSO-E and RTE have to be downloaded manually.

2D-distribution of price and production

This figure is obtained by running scripts/multiplots/main_scatter_price_production.py. The data from ENTSO-E and RTE have to be downloaded manually.

2D-distribution of price and weather

This figure is obtained by running scripts/multiplots/main_scatter_price_weather.py. The data from ENTSO-E have to be downloaded manually.

Description of the code

Documentation

An html documentation generated with Sphinx can be found in ./doc/pub_data_visualization.html.

Parameters of the scripts