data.datasets.sanity_checks.sanitycheck_eGon2035_electricity() - Code Metrics - Inspection of "Add sanity checks cts loads" - openego/eGon-data - Measure and Improve Code Quality continuously with Scrutinizer

Passed

Pull Request — dev (#924)

unknown

created 2022-09-09 14:41 UTC

sanitycheck_eGon2035_electricity() F

↳ Parent: data.datasets.sanity_checks

Complexity

Conditions

Size

Total Lines	230
Code Lines	97

Duplication

Lines	64
Ratio	27.83 %

Importance

Changes

Metric	Value
eloc	97
dl	64
loc	230
rs	1.7781
c	0
b	0
f	0
cc	16
nop	0

How to fix Long Method Complexity

Long Method

Small methods make your code easier to understand, in particular if combined with a good name. Besides, if your method is small, finding a good name is usually much easier.

For example, if you find yourself adding comments to a method's body, this is usually a good sign to extract the commented part to a new method, and use the comment as a starting point when coming up with a good name for this new method.

Commonly applied refactorings include:

If many parameters/temporary variables are present:
- Replace temporary variables with Query
- Introduce parameter object; often combined with preserve whole object
- If the above two are insufficient: Replace method with method object
If you have long conditionals: Decompose Conditional
Otherwise: Extract method

Complexity

Complex classes like data.datasets.sanity_checks.sanitycheck_eGon2035_electricity() often do a lot of different things. To break such a class down, we need to identify a cohesive component within that class. A common approach to find such a component is to look for fields/methods that share the same prefixes, or suffixes.

Once you have determined the fields that belong together, you can apply the Extract Class refactoring. If the component makes sense as a sub-class, Extract Subclass is also a candidate, and is often faster.

"""
This module does sanity checks for both the eGon2035 and the eGon100RE scenario
separately where a percentage error is given to showcase difference in output
and input values. Please note that there are missing input technologies in the
supply tables.
Authors: @ALonso, @dana
"""

import numpy as np
import pandas as pd

from egon.data import db, logger
from egon.data.datasets import Dataset
from egon.data.datasets.electricity_demand_timeseries.cts_buildings import (
    EgonCtsElectricityDemandBuildingShare,
    EgonCtsHeatDemandBuildingShare,
)


class SanityChecks(Dataset):
    def __init__(self, dependencies):
        super().__init__(
            name="SanityChecks",
            version="0.0.5",
            dependencies=dependencies,
            tasks={
                etrago_eGon2035_electricity,
                etrago_eGon2035_heat,
                residential_electricity_annual_sum,
                residential_electricity_hh_refinement,
                cts_electricity_demand_share,
                cts_heat_demand_share,
            },
        )


def etrago_eGon2035_electricity():
    """Execute basic sanity checks.

    Returns print statements as sanity checks for the electricity sector in
    the eGon2035 scenario.

    Parameters
    ----------
    None

    Returns
    -------
    None
    """

    scn = "eGon2035"

    # Section to check generator capacities
    logger.info(f"Sanity checks for scenario {scn}")
    logger.info(
        "For German electricity generators the following deviations between "
        "the inputs and outputs can be observed:"
    )

    carriers_electricity = [
        "other_non_renewable",
        "other_renewable",
        "reservoir",
        "run_of_river",
        "oil",
        "wind_onshore",
        "wind_offshore",
        "solar",
        "solar_rooftop",
        "biomass",
    ]

    for carrier in carriers_electricity:

        if carrier == "biomass":
            sum_output = db.select_dataframe(
                """SELECT scn_name, SUM(p_nom::numeric) as output_capacity_mw
                    FROM grid.egon_etrago_generator
                    WHERE bus IN (
                        SELECT bus_id FROM grid.egon_etrago_bus
                        WHERE scn_name = 'eGon2035'
                        AND country = 'DE')
                    AND carrier IN ('biomass', 'industrial_biomass_CHP',
                    'central_biomass_CHP')
                    GROUP BY (scn_name);
                """,
                warning=False,
            )

        else:
            sum_output = db.select_dataframe(
                f"""SELECT scn_name,
                 SUM(p_nom::numeric) as output_capacity_mw
                         FROM grid.egon_etrago_generator
                         WHERE scn_name = '{scn}'
                         AND carrier IN ('{carrier}')
                         AND bus IN
                             (SELECT bus_id
                               FROM grid.egon_etrago_bus
                               WHERE scn_name = 'eGon2035'
                               AND country = 'DE')
                         GROUP BY (scn_name);
                    """,
                warning=False,
            )

        sum_input = db.select_dataframe(
            f"""SELECT carrier, SUM(capacity::numeric) as input_capacity_mw
                     FROM supply.egon_scenario_capacities
                     WHERE carrier= '{carrier}'
                     AND scenario_name ='{scn}'
                     GROUP BY (carrier);
                """,
            warning=False,
        )

        if (

            sum_output.output_capacity_mw.sum() == 0
            and sum_input.input_capacity_mw.sum() == 0
        ):
            logger.info(
                f"No capacity for carrier '{carrier}' needed to be"
                f" distributed. Everything is fine"
            )

        elif (
            sum_input.input_capacity_mw.sum() > 0
            and sum_output.output_capacity_mw.sum() == 0
        ):
            logger.info(
                f"Error: Capacity for carrier '{carrier}' was not distributed "
                f"at all!"
            )

        elif (
            sum_output.output_capacity_mw.sum() > 0
            and sum_input.input_capacity_mw.sum() == 0
        ):
            logger.info(
                f"Error: Eventhough no input capacity was provided for carrier"
                f"'{carrier}' a capacity got distributed!"
            )

        else:
            sum_input["error"] = (
                (sum_output.output_capacity_mw - sum_input.input_capacity_mw)
                / sum_input.input_capacity_mw
            ) * 100
            g = sum_input["error"].values[0]

            logger.info(f"{carrier}: " + str(round(g, 2)) + " %")

    # Section to check storage units

    logger.info(f"Sanity checks for scenario {scn}")
    logger.info(
        "For German electrical storage units the following deviations between"
        "the inputs and outputs can be observed:"
    )

    carriers_electricity = ["pumped_hydro"]

    for carrier in carriers_electricity:

        sum_output = db.select_dataframe(
            f"""SELECT scn_name, SUM(p_nom::numeric) as output_capacity_mw
                         FROM grid.egon_etrago_storage
                         WHERE scn_name = '{scn}'
                         AND carrier IN ('{carrier}')
                         AND bus IN
                             (SELECT bus_id
                               FROM grid.egon_etrago_bus
                               WHERE scn_name = 'eGon2035'
                               AND country = 'DE')
                         GROUP BY (scn_name);
                    """,
            warning=False,
        )

        sum_input = db.select_dataframe(
            f"""SELECT carrier, SUM(capacity::numeric) as input_capacity_mw
                     FROM supply.egon_scenario_capacities
                     WHERE carrier= '{carrier}'
                     AND scenario_name ='{scn}'
                     GROUP BY (carrier);
                """,
            warning=False,
        )

        if (

            sum_output.output_capacity_mw.sum() == 0
            and sum_input.input_capacity_mw.sum() == 0
        ):
            logger.info(
                f"No capacity for carrier '{carrier}' needed to be "
                f"distributed. Everything is fine"
            )

        elif (
            sum_input.input_capacity_mw.sum() > 0
            and sum_output.output_capacity_mw.sum() == 0
        ):
            logger.info(
                f"Error: Capacity for carrier '{carrier}' was not distributed"
                f" at all!"
            )

        elif (
            sum_output.output_capacity_mw.sum() > 0
            and sum_input.input_capacity_mw.sum() == 0
        ):
            logger.info(
                f"Error: Eventhough no input capacity was provided for carrier"
                f" '{carrier}' a capacity got distributed!"
            )

        else:
            sum_input["error"] = (
                (sum_output.output_capacity_mw - sum_input.input_capacity_mw)
                / sum_input.input_capacity_mw
            ) * 100
            g = sum_input["error"].values[0]

            logger.info(f"{carrier}: " + str(round(g, 2)) + " %")

    # Section to check loads

    logger.info(
        "For German electricity loads the following deviations between the"
        " input and output can be observed:"
    )

    output_demand = db.select_dataframe(
        """SELECT a.scn_name, a.carrier,  SUM((SELECT SUM(p)
        FROM UNNEST(b.p_set) p))/1000000::numeric as load_twh
            FROM grid.egon_etrago_load a
            JOIN grid.egon_etrago_load_timeseries b
            ON (a.load_id = b.load_id)
            JOIN grid.egon_etrago_bus c
            ON (a.bus=c.bus_id)
            AND b.scn_name = 'eGon2035'
            AND a.scn_name = 'eGon2035'
            AND a.carrier = 'AC'
            AND c.scn_name= 'eGon2035'
            AND c.country='DE'
            GROUP BY (a.scn_name, a.carrier);

    """,
        warning=False,
    )["load_twh"].values[0]

    input_cts_ind = db.select_dataframe(
        """SELECT scenario,
         SUM(demand::numeric/1000000) as demand_mw_regio_cts_ind
            FROM demand.egon_demandregio_cts_ind
            WHERE scenario= 'eGon2035'
            AND year IN ('2035')
            GROUP BY (scenario);

        """,
        warning=False,
    )["demand_mw_regio_cts_ind"].values[0]

    input_hh = db.select_dataframe(
        """SELECT scenario, SUM(demand::numeric/1000000) as demand_mw_regio_hh
            FROM demand.egon_demandregio_hh
            WHERE scenario= 'eGon2035'
            AND year IN ('2035')
            GROUP BY (scenario);
        """,
        warning=False,
    )["demand_mw_regio_hh"].values[0]

    input_demand = input_hh + input_cts_ind

    e = round((output_demand - input_demand) / input_demand, 2) * 100

    logger.info(f"electricity demand: {e} %")


def etrago_eGon2035_heat():
    """Execute basic sanity checks.

    Returns print statements as sanity checks for the heat sector in
    the eGon2035 scenario.

    Parameters
    ----------
    None

    Returns
    -------
    None
    """

    # Check input and output values for the carriers "other_non_renewable",
    # "other_renewable", "reservoir", "run_of_river" and "oil"

    scn = "eGon2035"

    # Section to check generator capacities
    logger.info(f"Sanity checks for scenario {scn}")
    logger.info(
        "For German heat demands the following deviations between the inputs"
        " and outputs can be observed:"
    )

    # Sanity checks for heat demand

    output_heat_demand = db.select_dataframe(
        """SELECT a.scn_name,
          (SUM(
          (SELECT SUM(p) FROM UNNEST(b.p_set) p))/1000000)::numeric as load_twh
            FROM grid.egon_etrago_load a
            JOIN grid.egon_etrago_load_timeseries b
            ON (a.load_id = b.load_id)
            JOIN grid.egon_etrago_bus c
            ON (a.bus=c.bus_id)
            AND b.scn_name = 'eGon2035'
            AND a.scn_name = 'eGon2035'
            AND c.scn_name= 'eGon2035'
            AND c.country='DE'
            AND a.carrier IN ('rural_heat', 'central_heat')
            GROUP BY (a.scn_name);
        """,
        warning=False,
    )["load_twh"].values[0]

    input_heat_demand = db.select_dataframe(
        """SELECT scenario, SUM(demand::numeric/1000000) as demand_mw_peta_heat
            FROM demand.egon_peta_heat
            WHERE scenario= 'eGon2035'
            GROUP BY (scenario);
        """,
        warning=False,
    )["demand_mw_peta_heat"].values[0]

    e_demand = (
        round((output_heat_demand - input_heat_demand) / input_heat_demand, 2)
        * 100
    )

    logger.info(f"heat demand: {e_demand} %")

    # Sanity checks for heat supply

    logger.info(
        "For German heat supplies the following deviations between the inputs "
        "and outputs can be observed:"
    )

    # Comparison for central heat pumps
    heat_pump_input = db.select_dataframe(
        """SELECT carrier, SUM(capacity::numeric) as Urban_central_heat_pump_mw
            FROM supply.egon_scenario_capacities
            WHERE carrier= 'urban_central_heat_pump'
            AND scenario_name IN ('eGon2035')
            GROUP BY (carrier);
        """,
        warning=False,
    )["urban_central_heat_pump_mw"].values[0]

    heat_pump_output = db.select_dataframe(
        """SELECT carrier, SUM(p_nom::numeric) as Central_heat_pump_mw
            FROM grid.egon_etrago_link
            WHERE carrier= 'central_heat_pump'
            AND scn_name IN ('eGon2035')
            GROUP BY (carrier);
    """,
        warning=False,
    )["central_heat_pump_mw"].values[0]

    e_heat_pump = (
        round((heat_pump_output - heat_pump_input) / heat_pump_output, 2) * 100
    )

    logger.info(f"'central_heat_pump': {e_heat_pump} % ")

    # Comparison for residential heat pumps

    input_residential_heat_pump = db.select_dataframe(
        """SELECT carrier, SUM(capacity::numeric) as residential_heat_pump_mw
            FROM supply.egon_scenario_capacities
            WHERE carrier= 'residential_rural_heat_pump'
            AND scenario_name IN ('eGon2035')
            GROUP BY (carrier);
        """,
        warning=False,
    )["residential_heat_pump_mw"].values[0]

    output_residential_heat_pump = db.select_dataframe(
        """SELECT carrier, SUM(p_nom::numeric) as rural_heat_pump_mw
            FROM grid.egon_etrago_link
            WHERE carrier= 'rural_heat_pump'
            AND scn_name IN ('eGon2035')
            GROUP BY (carrier);
    """,
        warning=False,
    )["rural_heat_pump_mw"].values[0]

    e_residential_heat_pump = (
        round(
            (output_residential_heat_pump - input_residential_heat_pump)
            / input_residential_heat_pump,
            2,
        )
        * 100
    )
    logger.info(f"'residential heat pumps': {e_residential_heat_pump} %")

    # Comparison for resistive heater
    resistive_heater_input = db.select_dataframe(
        """SELECT carrier,
         SUM(capacity::numeric) as Urban_central_resistive_heater_MW
            FROM supply.egon_scenario_capacities
            WHERE carrier= 'urban_central_resistive_heater'
            AND scenario_name IN ('eGon2035')
            GROUP BY (carrier);
        """,
        warning=False,
    )["urban_central_resistive_heater_mw"].values[0]

    resistive_heater_output = db.select_dataframe(
        """SELECT carrier, SUM(p_nom::numeric) as central_resistive_heater_MW
            FROM grid.egon_etrago_link
            WHERE carrier= 'central_resistive_heater'
            AND scn_name IN ('eGon2035')
            GROUP BY (carrier);
        """,
        warning=False,
    )["central_resistive_heater_mw"].values[0]

    e_resistive_heater = (
        round(
            (resistive_heater_output - resistive_heater_input)
            / resistive_heater_input,
            2,
        )
        * 100
    )

    logger.info(f"'resistive heater': {e_resistive_heater} %")

    # Comparison for solar thermal collectors

    input_solar_thermal = db.select_dataframe(
        """SELECT carrier, SUM(capacity::numeric) as solar_thermal_collector_mw
            FROM supply.egon_scenario_capacities
            WHERE carrier= 'urban_central_solar_thermal_collector'
            AND scenario_name IN ('eGon2035')
            GROUP BY (carrier);
        """,
        warning=False,
    )["solar_thermal_collector_mw"].values[0]

    output_solar_thermal = db.select_dataframe(
        """SELECT carrier, SUM(p_nom::numeric) as solar_thermal_collector_mw
            FROM grid.egon_etrago_generator
            WHERE carrier= 'solar_thermal_collector'
            AND scn_name IN ('eGon2035')
            GROUP BY (carrier);
        """,
        warning=False,
    )["solar_thermal_collector_mw"].values[0]

    e_solar_thermal = (
        round(
            (output_solar_thermal - input_solar_thermal) / input_solar_thermal,
            2,
        )
        * 100
    )
    logger.info(f"'solar thermal collector': {e_solar_thermal} %")

    # Comparison for geothermal

    input_geo_thermal = db.select_dataframe(
        """SELECT carrier,
         SUM(capacity::numeric) as Urban_central_geo_thermal_MW
            FROM supply.egon_scenario_capacities
            WHERE carrier= 'urban_central_geo_thermal'
            AND scenario_name IN ('eGon2035')
            GROUP BY (carrier);
        """,
        warning=False,
    )["urban_central_geo_thermal_mw"].values[0]

    output_geo_thermal = db.select_dataframe(
        """SELECT carrier, SUM(p_nom::numeric) as geo_thermal_MW
            FROM grid.egon_etrago_generator
            WHERE carrier= 'geo_thermal'
            AND scn_name IN ('eGon2035')
            GROUP BY (carrier);
    """,
        warning=False,
    )["geo_thermal_mw"].values[0]

    e_geo_thermal = (
        round((output_geo_thermal - input_geo_thermal) / input_geo_thermal, 2)
        * 100
    )
    logger.info(f"'geothermal': {e_geo_thermal} %")


def residential_electricity_annual_sum(rtol=1e-5):
    """Sanity check for dataset electricity_demand_timeseries :
    Demand_Building_Assignment

    Aggregate the annual demand of all census cells at NUTS3 to compare
    with initial scaling parameters from DemandRegio.
    """

    df_nuts3_annual_sum = db.select_dataframe(
        sql="""
        SELECT dr.nuts3, dr.scenario, dr.demand_regio_sum, profiles.profile_sum
        FROM (
            SELECT scenario, SUM(demand) AS profile_sum, vg250_nuts3
            FROM demand.egon_demandregio_zensus_electricity AS egon,
             boundaries.egon_map_zensus_vg250 AS boundaries
            Where egon.zensus_population_id = boundaries.zensus_population_id
            AND sector = 'residential'
            GROUP BY vg250_nuts3, scenario
            ) AS profiles
        JOIN (
            SELECT nuts3, scenario, sum(demand) AS demand_regio_sum
            FROM demand.egon_demandregio_hh
            GROUP BY year, scenario, nuts3
              ) AS dr
        ON profiles.vg250_nuts3 = dr.nuts3 and profiles.scenario  = dr.scenario
        """
    )

    np.testing.assert_allclose(
        actual=df_nuts3_annual_sum["profile_sum"],
        desired=df_nuts3_annual_sum["demand_regio_sum"],
        rtol=rtol,
        verbose=False,
    )

    logger.info(
        "Aggregated annual residential electricity demand"
        " matches with DemandRegio at NUTS-3."
    )


def residential_electricity_hh_refinement(rtol=1e-5):
    """Sanity check for dataset electricity_demand_timeseries :
    Household Demands

    Check sum of aggregated household types after refinement method
    was applied and compare it to the original census values."""

    df_refinement = db.select_dataframe(
        sql="""
        SELECT refined.nuts3, refined.characteristics_code,
                refined.sum_refined::int, census.sum_census::int
        FROM(
            SELECT nuts3, characteristics_code, SUM(hh_10types) as sum_refined
            FROM society.egon_destatis_zensus_household_per_ha_refined
            GROUP BY nuts3, characteristics_code)
            AS refined
        JOIN(
            SELECT t.nuts3, t.characteristics_code, sum(orig) as sum_census
            FROM(
                SELECT nuts3, cell_id, characteristics_code,
                        sum(DISTINCT(hh_5types))as orig
                FROM society.egon_destatis_zensus_household_per_ha_refined
                GROUP BY cell_id, characteristics_code, nuts3) AS t
            GROUP BY t.nuts3, t.characteristics_code    ) AS census
        ON refined.nuts3 = census.nuts3
        AND refined.characteristics_code = census.characteristics_code
    """
    )

    np.testing.assert_allclose(
        actual=df_refinement["sum_refined"],
        desired=df_refinement["sum_census"],
        rtol=rtol,
        verbose=False,
    )

    logger.info("All Aggregated household types match at NUTS-3.")


def cts_electricity_demand_share(rtol=1e-5):
    """Sanity check for dataset electricity_demand_timeseries :
    CtsBuildings

    Check sum of aggregated cts electricity demand share which equals to one
    for every substation as the substation profile is linearly disaggregated
    to all buildings."""

    with db.session_scope() as session:
        cells_query = session.query(EgonCtsElectricityDemandBuildingShare)

    df_demand_share = pd.read_sql(
        cells_query.statement, cells_query.session.bind, index_col=None
    )

    np.testing.assert_allclose(
        actual=df_demand_share.groupby(["bus_id", "scenario"])[
            "profile_share"
        ].sum(),
        desired=1,
        rtol=rtol,
        verbose=False,
    )

    logger.info("The aggregated demand shares equal to one!.")


def cts_heat_demand_share(rtol=1e-5):
    """Sanity check for dataset electricity_demand_timeseries
    : CtsBuildings

    Check sum of aggregated cts heat demand share which equals to one
    for every substation as the substation profile is linearly disaggregated
    to all buildings."""

    with db.session_scope() as session:
        cells_query = session.query(EgonCtsHeatDemandBuildingShare)

    df_demand_share = pd.read_sql(
        cells_query.statement, cells_query.session.bind, index_col=None
    )

    np.testing.assert_allclose(
        actual=df_demand_share.groupby(["bus_id", "scenario"])[
            "profile_share"
        ].sum(),
        desired=1,
        rtol=rtol,
        verbose=False,
    )

    logger.info("The aggregated demand shares equal to one!.")


1		"""
2		This module does sanity checks for both the eGon2035 and the eGon100RE scenario
3		separately where a percentage error is given to showcase difference in output
4		and input values. Please note that there are missing input technologies in the
5		supply tables.
6		Authors: @ALonso, @dana
7		"""
8
9		import numpy as np
10		import pandas as pd
11
12		from egon.data import db, logger
13		from egon.data.datasets import Dataset
14		from egon.data.datasets.electricity_demand_timeseries.cts_buildings import (
15		EgonCtsElectricityDemandBuildingShare,
16		EgonCtsHeatDemandBuildingShare,
17		)
18
19
20		class SanityChecks(Dataset):
21		def __init__(self, dependencies):
22		super().__init__(
23		name="SanityChecks",
24		version="0.0.5",
25		dependencies=dependencies,
26		tasks={
27		etrago_eGon2035_electricity,
28		etrago_eGon2035_heat,
29		residential_electricity_annual_sum,
30		residential_electricity_hh_refinement,
31		cts_electricity_demand_share,
32		cts_heat_demand_share,
33		},
34		)
35
36
37		def etrago_eGon2035_electricity():
38		"""Execute basic sanity checks.
39
40		Returns print statements as sanity checks for the electricity sector in
41		the eGon2035 scenario.
42
43		Parameters
44		----------
45		None
46
47		Returns
48		-------
49		None
50		"""
51
52		scn = "eGon2035"
53
54		# Section to check generator capacities
55		logger.info(f"Sanity checks for scenario {scn}")
56		logger.info(
57		"For German electricity generators the following deviations between "
58		"the inputs and outputs can be observed:"
59		)
60
61		carriers_electricity = [
62		"other_non_renewable",
63		"other_renewable",
64		"reservoir",
65		"run_of_river",
66		"oil",
67		"wind_onshore",
68		"wind_offshore",
69		"solar",
70		"solar_rooftop",
71		"biomass",
72		]
73
74		for carrier in carriers_electricity:
75
76		if carrier == "biomass":
77		sum_output = db.select_dataframe(
78		"""SELECT scn_name, SUM(p_nom::numeric) as output_capacity_mw
79		FROM grid.egon_etrago_generator
80		WHERE bus IN (
81		SELECT bus_id FROM grid.egon_etrago_bus
82		WHERE scn_name = 'eGon2035'
83		AND country = 'DE')
84		AND carrier IN ('biomass', 'industrial_biomass_CHP',
85		'central_biomass_CHP')
86		GROUP BY (scn_name);
87		""",
88		warning=False,
89		)
90
91		else:
92		sum_output = db.select_dataframe(
93		f"""SELECT scn_name,
94		SUM(p_nom::numeric) as output_capacity_mw
95		FROM grid.egon_etrago_generator
96		WHERE scn_name = '{scn}'
97		AND carrier IN ('{carrier}')
98		AND bus IN
99		(SELECT bus_id
100		FROM grid.egon_etrago_bus
101		WHERE scn_name = 'eGon2035'
102		AND country = 'DE')
103		GROUP BY (scn_name);
104		""",
105		warning=False,
106		)
107
108		sum_input = db.select_dataframe(
109		f"""SELECT carrier, SUM(capacity::numeric) as input_capacity_mw
110		FROM supply.egon_scenario_capacities
111		WHERE carrier= '{carrier}'
112		AND scenario_name ='{scn}'
113		GROUP BY (carrier);
114		""",
115		warning=False,
116		)
117
118	View Code Duplication	if (
		0 ignored issues – show Duplication introduced 2022-06-03 07:55 UTC by Report Bug Copy Issue Report This code seems to be duplicated in your project. Loading history...
119		sum_output.output_capacity_mw.sum() == 0
120		and sum_input.input_capacity_mw.sum() == 0
121		):
122		logger.info(
123		f"No capacity for carrier '{carrier}' needed to be"
124		f" distributed. Everything is fine"
125		)
126
127		elif (
128		sum_input.input_capacity_mw.sum() > 0
129		and sum_output.output_capacity_mw.sum() == 0
130		):
131		logger.info(
132		f"Error: Capacity for carrier '{carrier}' was not distributed "
133		f"at all!"
134		)
135
136		elif (
137		sum_output.output_capacity_mw.sum() > 0
138		and sum_input.input_capacity_mw.sum() == 0
139		):
140		logger.info(
141		f"Error: Eventhough no input capacity was provided for carrier"
142		f"'{carrier}' a capacity got distributed!"
143		)
144
145		else:
146		sum_input["error"] = (
147		(sum_output.output_capacity_mw - sum_input.input_capacity_mw)
148		/ sum_input.input_capacity_mw
149		) * 100
150		g = sum_input["error"].values[0]
151
152		logger.info(f"{carrier}: " + str(round(g, 2)) + " %")
153
154		# Section to check storage units
155
156		logger.info(f"Sanity checks for scenario {scn}")
157		logger.info(
158		"For German electrical storage units the following deviations between"
159		"the inputs and outputs can be observed:"
160		)
161
162		carriers_electricity = ["pumped_hydro"]
163
164		for carrier in carriers_electricity:
165
166		sum_output = db.select_dataframe(
167		f"""SELECT scn_name, SUM(p_nom::numeric) as output_capacity_mw
168		FROM grid.egon_etrago_storage
169		WHERE scn_name = '{scn}'
170		AND carrier IN ('{carrier}')
171		AND bus IN
172		(SELECT bus_id
173		FROM grid.egon_etrago_bus
174		WHERE scn_name = 'eGon2035'
175		AND country = 'DE')
176		GROUP BY (scn_name);
177		""",
178		warning=False,
179		)
180
181		sum_input = db.select_dataframe(
182		f"""SELECT carrier, SUM(capacity::numeric) as input_capacity_mw
183		FROM supply.egon_scenario_capacities
184		WHERE carrier= '{carrier}'
185		AND scenario_name ='{scn}'
186		GROUP BY (carrier);
187		""",
188		warning=False,
189		)
190
191	View Code Duplication	if (
		0 ignored issues – show Duplication introduced 2022-06-03 07:55 UTC by Report Bug Copy Issue Report This code seems to be duplicated in your project. Loading history...
192		sum_output.output_capacity_mw.sum() == 0
193		and sum_input.input_capacity_mw.sum() == 0
194		):
195		logger.info(
196		f"No capacity for carrier '{carrier}' needed to be "
197		f"distributed. Everything is fine"
198		)
199
200		elif (
201		sum_input.input_capacity_mw.sum() > 0
202		and sum_output.output_capacity_mw.sum() == 0
203		):
204		logger.info(
205		f"Error: Capacity for carrier '{carrier}' was not distributed"
206		f" at all!"
207		)
208
209		elif (
210		sum_output.output_capacity_mw.sum() > 0
211		and sum_input.input_capacity_mw.sum() == 0
212		):
213		logger.info(
214		f"Error: Eventhough no input capacity was provided for carrier"
215		f" '{carrier}' a capacity got distributed!"
216		)
217
218		else:
219		sum_input["error"] = (
220		(sum_output.output_capacity_mw - sum_input.input_capacity_mw)
221		/ sum_input.input_capacity_mw
222		) * 100
223		g = sum_input["error"].values[0]
224
225		logger.info(f"{carrier}: " + str(round(g, 2)) + " %")
226
227		# Section to check loads
228
229		logger.info(
230		"For German electricity loads the following deviations between the"
231		" input and output can be observed:"
232		)
233
234		output_demand = db.select_dataframe(
235		"""SELECT a.scn_name, a.carrier, SUM((SELECT SUM(p)
236		FROM UNNEST(b.p_set) p))/1000000::numeric as load_twh
237		FROM grid.egon_etrago_load a
238		JOIN grid.egon_etrago_load_timeseries b
239		ON (a.load_id = b.load_id)
240		JOIN grid.egon_etrago_bus c
241		ON (a.bus=c.bus_id)
242		AND b.scn_name = 'eGon2035'
243		AND a.scn_name = 'eGon2035'
244		AND a.carrier = 'AC'
245		AND c.scn_name= 'eGon2035'
246		AND c.country='DE'
247		GROUP BY (a.scn_name, a.carrier);
248
249		""",
250		warning=False,
251		)["load_twh"].values[0]
252
253		input_cts_ind = db.select_dataframe(
254		"""SELECT scenario,
255		SUM(demand::numeric/1000000) as demand_mw_regio_cts_ind
256		FROM demand.egon_demandregio_cts_ind
257		WHERE scenario= 'eGon2035'
258		AND year IN ('2035')
259		GROUP BY (scenario);
260
261		""",
262		warning=False,
263		)["demand_mw_regio_cts_ind"].values[0]
264
265		input_hh = db.select_dataframe(
266		"""SELECT scenario, SUM(demand::numeric/1000000) as demand_mw_regio_hh
267		FROM demand.egon_demandregio_hh
268		WHERE scenario= 'eGon2035'
269		AND year IN ('2035')
270		GROUP BY (scenario);
271		""",
272		warning=False,
273		)["demand_mw_regio_hh"].values[0]
274
275		input_demand = input_hh + input_cts_ind
276
277		e = round((output_demand - input_demand) / input_demand, 2) * 100
278
279		logger.info(f"electricity demand: {e} %")
280
281
282		def etrago_eGon2035_heat():
283		"""Execute basic sanity checks.
284
285		Returns print statements as sanity checks for the heat sector in
286		the eGon2035 scenario.
287
288		Parameters
289		----------
290		None
291
292		Returns
293		-------
294		None
295		"""
296
297		# Check input and output values for the carriers "other_non_renewable",
298		# "other_renewable", "reservoir", "run_of_river" and "oil"
299
300		scn = "eGon2035"
301
302		# Section to check generator capacities
303		logger.info(f"Sanity checks for scenario {scn}")
304		logger.info(
305		"For German heat demands the following deviations between the inputs"
306		" and outputs can be observed:"
307		)
308
309		# Sanity checks for heat demand
310
311		output_heat_demand = db.select_dataframe(
312		"""SELECT a.scn_name,
313		(SUM(
314		(SELECT SUM(p) FROM UNNEST(b.p_set) p))/1000000)::numeric as load_twh
315		FROM grid.egon_etrago_load a
316		JOIN grid.egon_etrago_load_timeseries b
317		ON (a.load_id = b.load_id)
318		JOIN grid.egon_etrago_bus c
319		ON (a.bus=c.bus_id)
320		AND b.scn_name = 'eGon2035'
321		AND a.scn_name = 'eGon2035'
322		AND c.scn_name= 'eGon2035'
323		AND c.country='DE'
324		AND a.carrier IN ('rural_heat', 'central_heat')
325		GROUP BY (a.scn_name);
326		""",
327		warning=False,
328		)["load_twh"].values[0]
329
330		input_heat_demand = db.select_dataframe(
331		"""SELECT scenario, SUM(demand::numeric/1000000) as demand_mw_peta_heat
332		FROM demand.egon_peta_heat
333		WHERE scenario= 'eGon2035'
334		GROUP BY (scenario);
335		""",
336		warning=False,
337		)["demand_mw_peta_heat"].values[0]
338
339		e_demand = (
340		round((output_heat_demand - input_heat_demand) / input_heat_demand, 2)
341		* 100
342		)
343
344		logger.info(f"heat demand: {e_demand} %")
345
346		# Sanity checks for heat supply
347
348		logger.info(
349		"For German heat supplies the following deviations between the inputs "
350		"and outputs can be observed:"
351		)
352
353		# Comparison for central heat pumps
354		heat_pump_input = db.select_dataframe(
355		"""SELECT carrier, SUM(capacity::numeric) as Urban_central_heat_pump_mw
356		FROM supply.egon_scenario_capacities
357		WHERE carrier= 'urban_central_heat_pump'
358		AND scenario_name IN ('eGon2035')
359		GROUP BY (carrier);
360		""",
361		warning=False,
362		)["urban_central_heat_pump_mw"].values[0]
363
364		heat_pump_output = db.select_dataframe(
365		"""SELECT carrier, SUM(p_nom::numeric) as Central_heat_pump_mw
366		FROM grid.egon_etrago_link
367		WHERE carrier= 'central_heat_pump'
368		AND scn_name IN ('eGon2035')
369		GROUP BY (carrier);
370		""",
371		warning=False,
372		)["central_heat_pump_mw"].values[0]
373
374		e_heat_pump = (
375		round((heat_pump_output - heat_pump_input) / heat_pump_output, 2) * 100
376		)
377
378		logger.info(f"'central_heat_pump': {e_heat_pump} % ")
379
380		# Comparison for residential heat pumps
381
382		input_residential_heat_pump = db.select_dataframe(
383		"""SELECT carrier, SUM(capacity::numeric) as residential_heat_pump_mw
384		FROM supply.egon_scenario_capacities
385		WHERE carrier= 'residential_rural_heat_pump'
386		AND scenario_name IN ('eGon2035')
387		GROUP BY (carrier);
388		""",
389		warning=False,
390		)["residential_heat_pump_mw"].values[0]
391
392		output_residential_heat_pump = db.select_dataframe(
393		"""SELECT carrier, SUM(p_nom::numeric) as rural_heat_pump_mw
394		FROM grid.egon_etrago_link
395		WHERE carrier= 'rural_heat_pump'
396		AND scn_name IN ('eGon2035')
397		GROUP BY (carrier);
398		""",
399		warning=False,
400		)["rural_heat_pump_mw"].values[0]
401
402		e_residential_heat_pump = (
403		round(
404		(output_residential_heat_pump - input_residential_heat_pump)
405		/ input_residential_heat_pump,
406		2,
407		)
408		* 100
409		)
410		logger.info(f"'residential heat pumps': {e_residential_heat_pump} %")
411
412		# Comparison for resistive heater
413		resistive_heater_input = db.select_dataframe(
414		"""SELECT carrier,
415		SUM(capacity::numeric) as Urban_central_resistive_heater_MW
416		FROM supply.egon_scenario_capacities
417		WHERE carrier= 'urban_central_resistive_heater'
418		AND scenario_name IN ('eGon2035')
419		GROUP BY (carrier);
420		""",
421		warning=False,
422		)["urban_central_resistive_heater_mw"].values[0]
423
424		resistive_heater_output = db.select_dataframe(
425		"""SELECT carrier, SUM(p_nom::numeric) as central_resistive_heater_MW
426		FROM grid.egon_etrago_link
427		WHERE carrier= 'central_resistive_heater'
428		AND scn_name IN ('eGon2035')
429		GROUP BY (carrier);
430		""",
431		warning=False,
432		)["central_resistive_heater_mw"].values[0]
433
434		e_resistive_heater = (
435		round(
436		(resistive_heater_output - resistive_heater_input)
437		/ resistive_heater_input,
438		2,
439		)
440		* 100
441		)
442
443		logger.info(f"'resistive heater': {e_resistive_heater} %")
444
445		# Comparison for solar thermal collectors
446
447		input_solar_thermal = db.select_dataframe(
448		"""SELECT carrier, SUM(capacity::numeric) as solar_thermal_collector_mw
449		FROM supply.egon_scenario_capacities
450		WHERE carrier= 'urban_central_solar_thermal_collector'
451		AND scenario_name IN ('eGon2035')
452		GROUP BY (carrier);
453		""",
454		warning=False,
455		)["solar_thermal_collector_mw"].values[0]
456
457		output_solar_thermal = db.select_dataframe(
458		"""SELECT carrier, SUM(p_nom::numeric) as solar_thermal_collector_mw
459		FROM grid.egon_etrago_generator
460		WHERE carrier= 'solar_thermal_collector'
461		AND scn_name IN ('eGon2035')
462		GROUP BY (carrier);
463		""",
464		warning=False,
465		)["solar_thermal_collector_mw"].values[0]
466
467		e_solar_thermal = (
468		round(
469		(output_solar_thermal - input_solar_thermal) / input_solar_thermal,
470		2,
471		)
472		* 100
473		)
474		logger.info(f"'solar thermal collector': {e_solar_thermal} %")
475
476		# Comparison for geothermal
477
478		input_geo_thermal = db.select_dataframe(
479		"""SELECT carrier,
480		SUM(capacity::numeric) as Urban_central_geo_thermal_MW
481		FROM supply.egon_scenario_capacities
482		WHERE carrier= 'urban_central_geo_thermal'
483		AND scenario_name IN ('eGon2035')
484		GROUP BY (carrier);
485		""",
486		warning=False,
487		)["urban_central_geo_thermal_mw"].values[0]
488
489		output_geo_thermal = db.select_dataframe(
490		"""SELECT carrier, SUM(p_nom::numeric) as geo_thermal_MW
491		FROM grid.egon_etrago_generator
492		WHERE carrier= 'geo_thermal'
493		AND scn_name IN ('eGon2035')
494		GROUP BY (carrier);
495		""",
496		warning=False,
497		)["geo_thermal_mw"].values[0]
498
499		e_geo_thermal = (
500		round((output_geo_thermal - input_geo_thermal) / input_geo_thermal, 2)
501		* 100
502		)
503		logger.info(f"'geothermal': {e_geo_thermal} %")
504
505
506		def residential_electricity_annual_sum(rtol=1e-5):
507		"""Sanity check for dataset electricity_demand_timeseries :
508		Demand_Building_Assignment
509
510		Aggregate the annual demand of all census cells at NUTS3 to compare
511		with initial scaling parameters from DemandRegio.
512		"""
513
514		df_nuts3_annual_sum = db.select_dataframe(
515		sql="""
516		SELECT dr.nuts3, dr.scenario, dr.demand_regio_sum, profiles.profile_sum
517		FROM (
518		SELECT scenario, SUM(demand) AS profile_sum, vg250_nuts3
519		FROM demand.egon_demandregio_zensus_electricity AS egon,
520		boundaries.egon_map_zensus_vg250 AS boundaries
521		Where egon.zensus_population_id = boundaries.zensus_population_id
522		AND sector = 'residential'
523		GROUP BY vg250_nuts3, scenario
524		) AS profiles
525		JOIN (
526		SELECT nuts3, scenario, sum(demand) AS demand_regio_sum
527		FROM demand.egon_demandregio_hh
528		GROUP BY year, scenario, nuts3
529		) AS dr
530		ON profiles.vg250_nuts3 = dr.nuts3 and profiles.scenario = dr.scenario
531		"""
532		)
533
534		np.testing.assert_allclose(
535		actual=df_nuts3_annual_sum["profile_sum"],
536		desired=df_nuts3_annual_sum["demand_regio_sum"],
537		rtol=rtol,
538		verbose=False,
539		)
540
541		logger.info(
542		"Aggregated annual residential electricity demand"
543		" matches with DemandRegio at NUTS-3."
544		)
545
546
547		def residential_electricity_hh_refinement(rtol=1e-5):
548		"""Sanity check for dataset electricity_demand_timeseries :
549		Household Demands
550
551		Check sum of aggregated household types after refinement method
552		was applied and compare it to the original census values."""
553
554		df_refinement = db.select_dataframe(
555		sql="""
556		SELECT refined.nuts3, refined.characteristics_code,
557		refined.sum_refined::int, census.sum_census::int
558		FROM(
559		SELECT nuts3, characteristics_code, SUM(hh_10types) as sum_refined
560		FROM society.egon_destatis_zensus_household_per_ha_refined
561		GROUP BY nuts3, characteristics_code)
562		AS refined
563		JOIN(
564		SELECT t.nuts3, t.characteristics_code, sum(orig) as sum_census
565		FROM(
566		SELECT nuts3, cell_id, characteristics_code,
567		sum(DISTINCT(hh_5types))as orig
568		FROM society.egon_destatis_zensus_household_per_ha_refined
569		GROUP BY cell_id, characteristics_code, nuts3) AS t
570		GROUP BY t.nuts3, t.characteristics_code ) AS census
571		ON refined.nuts3 = census.nuts3
572		AND refined.characteristics_code = census.characteristics_code
573		"""
574		)
575
576		np.testing.assert_allclose(
577		actual=df_refinement["sum_refined"],
578		desired=df_refinement["sum_census"],
579		rtol=rtol,
580		verbose=False,
581		)
582
583		logger.info("All Aggregated household types match at NUTS-3.")
584
585
586		def cts_electricity_demand_share(rtol=1e-5):
587		"""Sanity check for dataset electricity_demand_timeseries :
588		CtsBuildings
589
590		Check sum of aggregated cts electricity demand share which equals to one
591		for every substation as the substation profile is linearly disaggregated
592		to all buildings."""
593
594		with db.session_scope() as session:
595		cells_query = session.query(EgonCtsElectricityDemandBuildingShare)
596
597		df_demand_share = pd.read_sql(
598		cells_query.statement, cells_query.session.bind, index_col=None
599		)
600
601		np.testing.assert_allclose(
602		actual=df_demand_share.groupby(["bus_id", "scenario"])[
603		"profile_share"
604		].sum(),
605		desired=1,
606		rtol=rtol,
607		verbose=False,
608		)
609
610		logger.info("The aggregated demand shares equal to one!.")
611
612
613		def cts_heat_demand_share(rtol=1e-5):
614		"""Sanity check for dataset electricity_demand_timeseries
615		: CtsBuildings
616
617		Check sum of aggregated cts heat demand share which equals to one
618		for every substation as the substation profile is linearly disaggregated
619		to all buildings."""
620
621		with db.session_scope() as session:
622		cells_query = session.query(EgonCtsHeatDemandBuildingShare)
623
624		df_demand_share = pd.read_sql(
625		cells_query.statement, cells_query.session.bind, index_col=None
626		)
627
628		np.testing.assert_allclose(
629		actual=df_demand_share.groupby(["bus_id", "scenario"])[
630		"profile_share"
631		].sum(),
632		desired=1,
633		rtol=rtol,
634		verbose=False,
635		)
636
637		logger.info("The aggregated demand shares equal to one!.")
638

openego / eGon-data

Pull Request — dev (#924)

sanitycheck_eGon2035_electricity() F

Complexity

Size

Duplication

Importance

How to fix Long Method Complexity

Long Method

Complexity

Duplication Side-by-Side

Filter issues like