Changelog

Unreleased

Added

Changed

Bug Fixes

Version 2.0.0 (2025-08-20)

Added

Changed

  • Update Apache Airflow version #36

  • Python 3.7 is not supported any more #118

  • update OSM version to 2023 #167

  • update MaStR version to 2023 #172

  • Update data-bundle #315

Bug Fixes

  • Fix entsoe-py dependency problems #116

  • Fix lines with resistance = 0 #120

  • Fix uncompatibilities with new Pandas versions >2.0.0 #124 #160

  • Fix unavailability of demand-regio with a provisional workaround #180

Version 1.0.0 (2025-05-15)

Added

  • Include description of the egon-data workflow in our documentation #23

  • There’s now a wrapper around subprocess.run in egon.data.subprocess.run. This wrapper catches errors better and displays better error messages than Python’s built-in function. Use this wrapper wenn calling other programs in Airflow tasks.

  • You can now override the default database configuration using command line arguments. Look for the switches starting with --database in egon-data --help. See PR #159 for more details.

  • Docker will not be used if there is already a service listening on the HOST:PORT combination configured for the database.

  • You can now supply values for the command line arguments for egon-data using a configuration file. If the configuration file doesn’t exist, it will be created by egon-data on it’s first run. Note that the configuration file is read from and written to the directtory in which egon-data is started, so it’s probably best to run egon-data in a dedicated directory. There’s also the new function egon.data.config.settings which returns the current configuration settings. See PR #159 for more details.

  • You can now use tasks which are not part of a Dataset, i.e. which are unversioned, as dependencies of a dataset. See #318 for more details.

  • You can now force the tasks of a Dataset to be always executed by giving the version of the Dataset a ".dev" suffix. See #318 for more details.

  • OSM data import as done in open_ego #1 which was updated to the latest long-term data set of the 2021-01-01 in #223

  • Verwaltungsgebiete data import (vg250) more or less done as in open_ego #3

  • Zensus population data import #2

  • Zensus data import for households, apartments and buildings #91

  • DemandRegio data import for annual electricity demands #5

  • Download cleaned open-MaStR data from Zenodo #14

  • NEP 2021 input data import #45

  • Option for running workflow in test mode #112

  • Abstraction of hvmv and ehv substations #9

  • Filter zensus being inside Germany and assign population to municipalities #7

  • RE potential areas data import #124

  • Heat demand data import #101

  • Demographic change integration #47

  • Creation of voronoi polygons for hvmv and ehv substations #9

  • Add hydro and biomass power plants eGon2035 #127

  • Creation of the ehv/hv grid model with osmTGmod, see issue #4 and PR #164

  • Identification of medium-voltage grid districts #10

  • Distribute electrical demands of households to zensus cells #181

  • Distribute electrical demands of cts to zensus cells #210

  • Include industrial sites’ download, import and merge #117

  • Integrate scenario table with parameters for each sector #177

  • The volume of the docker container for the PostgreSQL database is saved in the project directory under docker/database-data. The current user ($USER) is owner of the volume. Containers created prior to this change will fail when using the changed code. The container needs to be re-created. #228

  • Extract landuse areas from OSM #214

  • Integrate weather data and renewable feedin timeseries #19

  • Create and import district heating areas #162

  • Integrate electrical load time series for cts sector #109

  • Assign voltage level and bus_id to power plants #15

  • Integrate solar rooftop for etrago tables #255

  • Integrate gas bus and link tables #198

  • Integrate data bundle #272

  • Add household electricity demand time series, mapping of demand profiles to census cells and aggregated household electricity demand time series at MV grid district level #256

  • Integrate power-to-gas installation potential links #293

  • Integrate distribution of wind onshore and pv ground mounted generation #146

  • Integrate dynamic line rating potentials #72

  • Integrate gas voronoi polygons #308

  • Integrate supply strategies for individual and district heating #232

  • Integrate gas production #321

  • Integrate industrial time series creation #237

  • Merge electrical loads per bus and export to etrago tables #328

  • Insert industial gas demand #358

  • Integrate existing CHP and extdended CHP > 10MW_el #266

  • Add random seed to CLI parameters #351

  • Extend zensus by a combined table with all cells where there’s either building, apartment or population data #359

  • Include allocation of pumped hydro units #332

  • Add example metadata for OSM, VG250 and Zensus VG250. Add metadata templates for licences, context and some helper functions. Extend docs on how to create metadata for tables. #139

  • Integrate DSM potentials for CTS and industry #259

  • Assign weather cell id to weather dependant power plants #330

  • Distribute wind offshore capacities #329

  • Add CH4 storages #405

  • Include allocation of conventional (non CHP) power plants #392

  • Fill egon-etrago-generators table #485

  • Include time-dependent coefficient of performance for heat pumps #532

  • Limit number of parallel processes per task #265

  • Include biomass CHP plants to eTraGo tables #498

  • Include Pypsa default values in table creation #544

  • Include PHS in eTraGo tables #333

  • Include feedin time series for wind offshore #531

  • Include carrier names in eTraGo table #551

  • Include hydrogen infrastructure for eGon2035 scenario #474

  • Include downloaded pypsa-eur-sec results #138

  • Create heat buses for eGon100RE scenario #582

  • Filter for DE in gas infrastructure deletion at beginning of respective tasks #567

  • Insert open cycle gas turbines into eTraGo tables #548

  • Preprocess buildings and amenities for LV grids #262

  • Assign household profiles to OSM buildings #435

  • Add link to meta creator to docs #599

  • Add extendable batteries and heat stores #566

  • Add efficiency, capital_cost and marginal_cost to gas related data in etrago tables #596

  • Add wind onshore farms for the eGon100RE scenario #690

  • The shared memory under “/dev/shm” is now shared between host and container. This was done because Docker has a rather tiny default for the size of “/dev/shm” which caused random problems. Guessing what size is correct is also not a good idea, so sharing between host and container seems like the best option. This restricts using egon-data with docker to Linux and MacOS, if the latter has “/dev/shm” but seems like the best course of action for now. Done via PR #703 and hopefully prevents issues #702 and `#267`_ from ever occurring again.

  • Provide wrapper to catch DB unique violation #514

  • Add electric scenario parameters for eGon100RE #699

  • Introduce Sanity checks for eGon2035 #382

  • Add motorized individual travel #553

  • Allocating MaStR PV rooftop power plants to OSM and synthetic buildings. Desaggregating PV rooftop scenarios to mv grid districts and OSM and synthetic buildings. #684

  • Add mapping zensus - weather cells #845

  • Add pv rooftop plants per mv grid for eGon100RE #861

  • Integrated heavy duty transport FCEV #552

  • Assign CTS demands to buildings #671

  • Add sanity checks for residential electricity loads #902

  • Add sanity checks for cts loads #919

  • Add distribution of CHP plants for eGon100RE #851

  • Add mapping table for all used buildings #962

  • Add charging infrastructure for e-mobility #937

  • Add zipfile check #969

  • Add marginal costs for generators abroad and for carriers nuclear and coal #907

  • Add wind off shore power plants for eGon100RE #868

  • Write simBEV metadata to DB table PR #978

  • Add voltage level for electricity building loads #955

  • Add desaggregation of pv home batteries onto buildings #988

  • Desaggregation of DSM time series onto CTS consumers per bus id and individual indutry consumers. #1048

  • Add load areas #1014

  • Add new MaStR dataset #1051

  • Heat pump desaggregation to buildings PR #903

  • Add low flex scenario ‘eGon2035_lowflex’ #822

  • Add Status2023 update heat demand #199

  • Add status2023 electrical neighbours #182

  • Add Status2023 to gas nodes #205

  • Update osm for status2023 #169

  • Add wrapped_partial to dynamise task generation #207

  • Add MaStR geocoding and handling of conventional generators #1095

  • Improve building household load assignment #350

  • Add CLI option to prefix DAG name in Airflow config #1252

Changed

  • Adapt structure of the documentation to project specific requirements #20

  • Switch from Travis to GitHub actions for CI jobs #92

  • Rename columns to id and zensus_population_id in zensus tables #140

  • Revise docs CONTRIBUTING section and in particular PR guidelines #88 and #145

  • Drop support for Python3.6 #148

  • Improve selection of zensus data in test mode #151

  • Delete tables before re-creation and data insertation #166

  • Adjust residential heat demand in unpopulated zenus cells #167

  • Introduce mapping between VG250 municipalities and census cells #165

  • Delete tables if they exist before re-creation and data insertation #166

  • Add gdal to pre-requisites #185

  • Update task zensus-inside-germany #196

  • Update installation of demandregio’s disaggregator #202

  • Update etrago tables #243 and #285

  • Migrate VG250 to datasets #283

  • Allow configuring the airflow port #281

  • Migrate mastr, mv_grid_districts and re_potential_areas to datasets #297

  • Migrate industrial sites to datasets #237

  • Rename etrago tables from e.g. egon_pf_hv_bus to egon_etrago bus etc. #334

  • Move functions used by multiple datasets #323

  • Migrate scenario tables to datasets #309

  • Migrate weather data and power plants to datasets #314

  • Create and fill table for CTS electricity demand per bus #326

  • Migrate osmTGmod to datasets #305

  • Filter osm landuse areas, rename industrial sites tables and update load curve function #378

  • Remove version columns from eTraGo tables and related code #384

  • Remove country column from scenario capacities table #391

  • Update version of zenodo download #397

  • Rename columns gid to id #169

  • Remove upper version limit of pandas #383

  • Use random seed from CLI parameters for CHP and society prognosis functions #351

  • Changed demand.egon_schmidt_industrial_sites - table and merged table (industrial_sites) #423

  • Replace ‘gas’ carrier with ‘CH4’ and ‘H2’ carriers #436

  • Adjust file path for industrial sites import #418

  • Rename columns subst_id to bus_id #335

  • Apply black and isort for all python scripts #463

  • Update deposit id for zenodo download #498

  • Add to etrago.setug.py the busmap table #484

  • Migrate dlr script to datasets #508

  • Migrate loadarea scripts to datasets #525

  • Migrate plot.py to dataset of district heating areas #527

  • Migrate substation scripts to datasets #304

  • Update deposit_id for zenodo download #540

  • Add household demand profiles to etrago table #381

  • Migrate zensus scripts to datasets #422

  • Add information on plz, city and federal state to data on mastr without chp #425

  • Assign residential heat demands to osm buildings #557

  • Add foreign gas buses and adjust cross bording pipelines #545

  • Integrate fuel and CO2 costs for eGon2035 to scenario parameters #549

  • Aggregate generators and stores for CH4 #629

  • Fill missing household data for populated cells #431

  • Fix RE potential areas outside of Germany by updating the dataset. Import files from data bundle. #592 #595

  • Add DC lines from Germany to Sweden and Denmark #611

  • H2 demand is met from the H2_grid buses. In Addtion, it can be met from the H2_saltcavern buses if a proximity criterion is fulfilled #620

  • Create H2 pipeline infrastructure for eGon100RE #638

  • Change refinement method for households types #651

  • H2 feed in links are changed to non extendable #653

  • Remove the ‘_fixed’ suffix #628

  • Fill table demand.egon_demandregio_zensus_electricity after profile allocation #586

  • Change method of building assignment #663

  • Create new OSM residential building table #587

  • Move python-operators out of pipeline #644

  • Add annualized investment costs to eTraGo tables #672

  • Improve modelling of NG and biomethane production #678

  • Unify carrier names for both scenarios #575

  • Add automatic filtering of gas data: Pipelines of length zero and gas buses isolated of the grid are deleted. #590

  • Add gas data in neighbouring countries #727

  • Aggregate DSM components per substation #661

  • Aggregate NUTS3 industrial loads for CH4 and H2 #452

  • Update OSM dataset from 2021-02-02 to 2022-01-01 #486

  • Update deposit id to access v0.6 of the zenodo repository #627

  • Include electricity storages for eGon100RE scenario #581

  • Update deposit id to access v0.7 of the zenodo repository #736

  • Include simplified restrictions for H2 feed-in into CH4 grid #790

  • Update hh electricity profiles #735

  • Improve CH4 stores and productions aggregation by removing dedicated task #775

  • Add CH4 stores in Germany for eGon100RE #779

  • Assigment of H2 and CH4 capacitites for pipelines in eGon100RE #686

  • Update deposit id to access v0.8 of the zenodo repository #760

  • Add primary key to table openstreetmap.osm_ways_with_segments #787

  • Update pypsa-eur-sec fork and store national demand time series #402

  • Move and merge the two assign_gas_bus_id functions to a central place #797

  • Add coordinates to non AC buses abroad in eGon100RE #803

  • Integrate additional industrial electricity demands for eGon100RE #817

  • Set non extendable gas components from p-e-s as so for eGon100RE #877

  • Integrate new data bundle using zenodo sandbox #866

  • Add noflex scenario for motorized individual travel #821

  • Allocate PV home batteries to mv grid districts #749

  • Add sanity checks for motorized individual travel #820

  • Parallelize sanity checks #882

  • Insert crossboarding gas pipeline with Germany in eGon100RE #881

  • Harmonize H2 carrier names in eGon100RE #929

  • Rename noflex to lowflex scenario for motorized individual travel #921

  • Update creation of heat demand timeseries #857 #856

  • Overwrite retrofitted_CH4pipeline-to-H2pipeline_share with pes result #933

  • Adjust H2 industry profiles abroad for eGon2035 #940

  • Introduce carrier name ‘others’ #819

  • Add rural heat pumps per medium voltage grid district #987

  • Add eGon2021 scenario to demandregio dataset #1035

  • Update MaStR dataset #519

  • Add missing VOM costs for heat sector components #942

  • Add sanity checks for gas sector in eGon2035 #864

  • Desaggregate industry demands to OSM areas and industrial sites #1001

  • Add gas generator in Norway #1074

  • SQLAlchemy engine objects created via egon.data.db.engine are now cached on a per process basis, so only one engine is ever created for a single process. This fixes issue #799.

  • Insert rural heat per supply technology #1026

  • Insert lifetime for components from p-e-s in eGon100RE #1073

  • Change hgv data source to use database #1086

  • Rename eMob MIT carrier names (use underscores) #1105

  • Change desposit ID for data_bundle download from zenodo sandbox #1110

  • Use MaStR geocoding results for pv rooftop to buildings mapping workflow #1095

  • Add eMob MIT for SQ2023 scenario #176

Bug Fixes

  • Some dependencies have their upper versions restricted now. This is mostly due to us not yet supporting Airflow 2.0 which means that it will no longer work with certain packages, but we also won’t get and upper version limit for those from Airflow because version 1.X is unlikely to to get an update. So we had to make some implicit dependencies explicit in order to give them them upper version limits. Done via PR #692 in order to fix issues `#343`_, #556, #641 and #669.

  • Heat demand data import #157

  • Substation sequence #171

  • Adjust names of demandregios nuts3 regions according to nuts version 2016 #201

  • Delete zensus buildings, apartments and households in unpopulated cells #202

  • Fix input table of electrical-demands-zensus #217

  • Import heat demand raster files successively to fix import for dataset==Everything #204

  • Replace wrong table name in SQL function used in substation extraction #236

  • Fix osmtgmod for osm data from 2021 by updating substation in Garenfeld and set srid #241 #258

  • Adjust format of voltage levels in hvmv substation #248

  • Change order of osmtgmod tasks #253

  • Fix missing municipalities #279

  • Fix import of hydro power plants #270

  • Fix path to osm-file for osmtgmod_osm_import #258

  • Fix conflicting docker containers by setting a project name #289

  • Update task insert-nep-data for pandas version 1.3.0 #322

  • Fix versioning conflict with mv_grid_districts #340

  • Set current working directory as java’s temp dir when executing osmosis #344

  • Fix border gas voronoi polygons which had no bus_id #362

  • Add dependency from WeatherData to Vg250 #387

  • Fix unnecessary columns in normal mode for inserting the gas production #390

  • Add xlrd and openpyxl to installation setup #400

  • Store files of OSM, zensus and VG250 in working dir #341

  • Remove hard-coded slashes in file paths to ensure Windows compatibility #398

  • Add missing dependency in pipeline.py #412

  • Add prefix egon to MV grid district tables #349

  • Bump MV grid district version no #432

  • Add curl to prerequisites in the docs #440

  • Replace NAN by 0 to avoid empty p_set column in DB #414

  • Exchange bus 0 and bus 1 in Power-to-H2 links #458

  • Fix missing cts demands for eGon2035 #511

  • Add data_bundle to industrial_sites task dependencies #468

  • Lift geopandas minimum requirement to 0.10.0 #504

  • Use inbuilt datetime package instead of pandas.datetime #516

  • Add missing ‘sign’ for CH4 and H2 loads #538

  • Delete only AC loads for eTraGo in electricity_demand_etrago #535

  • Filter target values by scenario name #570

  • Reduce number of timesteps of hh electricity demand profiles to 8760 #593

  • Fix assignemnt of heat demand profiles at German borders #585

  • Change source for H2 steel tank storage to Danish Energy Agency #605

  • Change carrier name from ‘pv’ to ‘solar’ in eTraGo_generators #617

  • Assign “carrier” to transmission lines with no value in grid.egon_etrago_line #625

  • Fix deleting from eTraGo tables #613

  • Fix positions of the foreign gas buses #618

  • Create and fill transfer_busses table in substation-dataset #610

  • H2 steel tanks are removed again from saltcavern storage #621

  • Timeseries not deleted from grid.etrago_generator_timeseries #645

  • Fix function to get scaled hh profiles #674

  • Change order of pypsa-eur-sec and scenario-capacities #589

  • Fix gas storages capacities #676

  • Distribute rural heat supply to residetntial and service demands #679

  • Fix time series creation for pv rooftop #688

  • Fix extraction of buildings without amenities #693

  • Assign DLR capacities to every transmission line #683

  • Fix solar ground mounted total installed capacity #695

  • Fix twisted number error residential demand #704

  • Fix industrial H2 and CH4 demand for eGon100RE scenario #687

  • Clean up “pipeline.py” #562

  • Assign timeseries data to crossborder generators ego2035 #724

  • Add missing dataset dependencies in “pipeline.py” #725

  • Fix assignemnt of impedances (x) to etrago tables #710

  • Fix country_code attribution of two gas buses #744

  • Fix voronoi assignemnt for enclaves #734

  • Set lengths of non-pipeline links to 0 #741

  • Change table name from boundaries.saltstructures_inspee to boundaries.inspee_saltstructures #746

  • Add missing marginal costs for conventional generators in Germany #722

  • Fix carrier name for solar ground mounted in scenario parameters #752

  • Create rural_heat buses only for mv grid districts with heat load #708

  • Solve problem while creating generators series data egon2035 #758

  • Correct wrong carrier name when assigning marginal costs #766

  • Use db.next_etrago_id in dsm and pv_rooftop dataset #748

  • Add missing dependency to heat_etrago #771

  • Fix country code of gas pipeline DE-AT #813

  • Fix distribution of resistive heaters in district heating grids #783

  • Fix missing reservoir and run_of_river power plants in eTraGo tables, Modify fill_etrago_gen to also group generators from eGon100RE, Use db.next_etrago_id in fill_etrago_gen #798 #776

  • Fix model load timeseries in motorized individual travel #830

  • Fix gas costs #847

  • Add imports that have been wrongly deleted #849

  • Fix final demand of heat demand timeseries #781

  • Add extendable batteries only to buses at substations #852

  • Move class definition for grid.egon_gas_voronoi out of etrago_setup #888

  • Temporarily set upper version limit for pandas #829

  • Change industrial gas load modelling #871

  • Delete eMob MIT data from eTraGo tables on init #878

  • Fix model id issues in DSM potentials for CTS and industry #901

  • Drop isolated buses and tranformers in eHV grid #874

  • Model gas turbines always as links #914

  • Drop era5 weather cell table using cascade #909

  • Remove drop of p_set and q_set for loads without timeserie #971

  • Delete gas bus with wrong country code #958

  • Overwrite capacities for conventional power plants with data from nep list #403

  • Make gas grid links bidirectional #1021

  • Correct gas technology costs for eGon100RE #984

  • Adjust p_nom and marginal cost for OCGT in eGon2035 #863

  • Mismatch of building bus_ids from cts_heat_demand_building_share and mapping table #989

  • Fix zensus weather cells mapping #1031

  • Fix solar rooftop in test mode #1055

  • Add missing filter for scenario name in chp expansion #1015

  • Fix installed capacity per individual heat pump #1058

  • Add missing gas turbines abroad #1079

  • Fix gas generators abroad (marginal cost and e_nom_max) #1075

  • Fix gas pipelines isolated of the German grid #1081

  • Fix aggregation of DSM-components #1069

  • Fix URL of TYNDP scenario dataset

  • Automatically generated tasks now get unique task_ids. Fixes issue #985 via PR #986.

  • Adjust capcities of German CH4 stores #1096

  • Fix faulty DSM time series #1088

  • Set upper limit on commissioning date for units from MaStR dataset #1098

  • Fix conversion factor for CH4 loads abroad in eGon2035 #1104

  • Change structure of documentation in rtd #1126

  • Fix URL of eGon data-bundle dataset #1154

  • Fix URLs of MaStR datasets

  • Fix CRS in ERA5 transformation #1159

  • Fix depricated python-operator import #183

  • Fix incorrect YAML serialization of the –scenarios CLI parameter that caused broken DAGs in the pipeline. #343

  • Fix hard coded scenario name in CTS building generation #1180

  • Update postgresql version but assure compatibility with osmosis #1185

  • Fix Zensus population download #1187

  • Fix DsmPotential dependency errors #1157

  • Fix incorrect handling of –scenarios CLI option by converting tuple to list #1192

  • Fix household profile allocation to buildings #1190