core.utilities.aggregate_hourly_data_by_period()   F
last analyzed

Complexity

Conditions 27

Size

Total Lines 147
Code Lines 119

Duplication

Lines 59
Ratio 40.14 %

Importance

Changes 0
Metric Value
cc 27
eloc 119
nop 4
dl 59
loc 147
rs 0
c 0
b 0
f 0

How to fix   Long Method    Complexity   

Long Method

Small methods make your code easier to understand, in particular if combined with a good name. Besides, if your method is small, finding a good name is usually much easier.

For example, if you find yourself adding comments to a method's body, this is usually a good sign to extract the commented part to a new method, and use the comment as a starting point when coming up with a good name for this new method.

Commonly applied refactorings include:

Complexity

Complex classes like core.utilities.aggregate_hourly_data_by_period() often do a lot of different things. To break such a class down, we need to identify a cohesive component within that class. A common approach to find such a component is to look for fields/methods that share the same prefixes, or suffixes.

Once you have determined the fields that belong together, you can apply the Extract Class refactoring. If the component makes sense as a sub-class, Extract Subclass is also a candidate, and is often faster.

1
from datetime import datetime, timedelta
2
import mysql.connector
3
import collections
4
from decimal import Decimal
5
import config
6
import statistics
7
8
9
########################################################################################################################
10
# Aggregate hourly data by period
11
# rows_hourly: list of (start_datetime_utc, actual_value), should belong to one energy_category_id
12
# start_datetime_utc: start datetime in utc
13
# end_datetime_utc: end datetime in utc
14
# period_type: one of the following period types, 'hourly', 'daily', 'monthly' and 'yearly'
15
# Note: this procedure doesn't work with multiple energy categories
16
########################################################################################################################
17
def aggregate_hourly_data_by_period(rows_hourly, start_datetime_utc, end_datetime_utc, period_type):
18
    # todo: validate parameters
19
    start_datetime_utc = start_datetime_utc.replace(tzinfo=None)
20
    end_datetime_utc = end_datetime_utc.replace(tzinfo=None)
21
22
    if period_type == "hourly":
23
        result_rows_hourly = list()
24
        # todo: add config.working_day_start_time_local
25
        # todo: add config.minutes_to_count
26
        current_datetime_utc = start_datetime_utc.replace(minute=0, second=0, microsecond=0, tzinfo=None)
27
        while current_datetime_utc <= end_datetime_utc:
28
            subtotal = Decimal(0.0)
29
            for row in rows_hourly:
30
                if current_datetime_utc <= row[0] < current_datetime_utc + \
31
                        timedelta(minutes=config.minutes_to_count):
32
                    subtotal += row[1]
33
            result_rows_hourly.append((current_datetime_utc, subtotal))
34
            current_datetime_utc += timedelta(minutes=config.minutes_to_count)
35
36
        return result_rows_hourly
37
38
    elif period_type == "daily":
39
        result_rows_daily = list()
40
        # todo: add config.working_day_start_time_local
41
        # todo: add config.minutes_to_count
42
        # calculate the start datetime in utc of the first day in local
43
        start_datetime_local = start_datetime_utc + timedelta(hours=int(config.utc_offset[1:3]))
44
        current_datetime_utc = start_datetime_local.replace(hour=0) - timedelta(hours=int(config.utc_offset[1:3]))
45
        while current_datetime_utc <= end_datetime_utc:
46
            subtotal = Decimal(0.0)
47
            for row in rows_hourly:
48
                if current_datetime_utc <= row[0] < current_datetime_utc + timedelta(days=1):
49
                    subtotal += row[1]
50
            result_rows_daily.append((current_datetime_utc, subtotal))
51
            current_datetime_utc += timedelta(days=1)
52
53
        return result_rows_daily
54
55
    elif period_type == "monthly":
56
        result_rows_monthly = list()
57
        # todo: add config.working_day_start_time_local
58
        # todo: add config.minutes_to_count
59
        # calculate the start datetime in utc of the first day in the first month in local
60
        start_datetime_local = start_datetime_utc + timedelta(hours=int(config.utc_offset[1:3]))
61
        current_datetime_utc = \
62
            start_datetime_local.replace(day=1, hour=0) - timedelta(hours=int(config.utc_offset[1:3]))
63
64
        while current_datetime_utc <= end_datetime_utc:
65
            # calculate the next datetime in utc
66 View Code Duplication
            if current_datetime_utc.month == 1:
0 ignored issues
show
Duplication introduced by
This code seems to be duplicated in your project.
Loading history...
67
                temp_day = 28
68
                ny = current_datetime_utc.year
69
                if (ny % 100 != 0 and ny % 4 == 0) or (ny % 100 == 0 and ny % 400 == 0):
70
                    temp_day = 29
71
72
                next_datetime_utc = datetime(year=current_datetime_utc.year,
73
                                             month=current_datetime_utc.month + 1,
74
                                             day=temp_day,
75
                                             hour=current_datetime_utc.hour,
76
                                             minute=current_datetime_utc.minute,
77
                                             second=0,
78
                                             microsecond=0,
79
                                             tzinfo=None)
80
            elif current_datetime_utc.month == 2:
81
                next_datetime_utc = datetime(year=current_datetime_utc.year,
82
                                             month=current_datetime_utc.month + 1,
83
                                             day=31,
84
                                             hour=current_datetime_utc.hour,
85
                                             minute=current_datetime_utc.minute,
86
                                             second=0,
87
                                             microsecond=0,
88
                                             tzinfo=None)
89
            elif current_datetime_utc.month in [3, 5, 8, 10]:
90
                next_datetime_utc = datetime(year=current_datetime_utc.year,
91
                                             month=current_datetime_utc.month + 1,
92
                                             day=30,
93
                                             hour=current_datetime_utc.hour,
94
                                             minute=current_datetime_utc.minute,
95
                                             second=0,
96
                                             microsecond=0,
97
                                             tzinfo=None)
98
            elif current_datetime_utc.month == 7:
99
                next_datetime_utc = datetime(year=current_datetime_utc.year,
100
                                             month=current_datetime_utc.month + 1,
101
                                             day=31,
102
                                             hour=current_datetime_utc.hour,
103
                                             minute=current_datetime_utc.minute,
104
                                             second=0,
105
                                             microsecond=0,
106
                                             tzinfo=None)
107
            elif current_datetime_utc.month in [4, 6, 9, 11]:
108
                next_datetime_utc = datetime(year=current_datetime_utc.year,
109
                                             month=current_datetime_utc.month + 1,
110
                                             day=31,
111
                                             hour=current_datetime_utc.hour,
112
                                             minute=current_datetime_utc.minute,
113
                                             second=0,
114
                                             microsecond=0,
115
                                             tzinfo=None)
116
            elif current_datetime_utc.month == 12:
117
                next_datetime_utc = datetime(year=current_datetime_utc.year + 1,
118
                                             month=1,
119
                                             day=31,
120
                                             hour=current_datetime_utc.hour,
121
                                             minute=current_datetime_utc.minute,
122
                                             second=0,
123
                                             microsecond=0,
124
                                             tzinfo=None)
125
126
            subtotal = Decimal(0.0)
127
            for row in rows_hourly:
128
                if current_datetime_utc <= row[0] < next_datetime_utc:
0 ignored issues
show
introduced by
The variable next_datetime_utc does not seem to be defined for all execution paths.
Loading history...
129
                    subtotal += row[1]
130
131
            result_rows_monthly.append((current_datetime_utc, subtotal))
132
            current_datetime_utc = next_datetime_utc
133
134
        return result_rows_monthly
135
136
    elif period_type == "yearly":
137
        result_rows_yearly = list()
138
        # todo: add config.working_day_start_time_local
139
        # todo: add config.minutes_to_count
140
        # calculate the start datetime in utc of the first day in the first month in local
141
        start_datetime_local = start_datetime_utc + timedelta(hours=int(config.utc_offset[1:3]))
142
        current_datetime_utc = start_datetime_local.replace(month=1, day=1, hour=0) - timedelta(
143
            hours=int(config.utc_offset[1:3]))
144
145
        while current_datetime_utc <= end_datetime_utc:
146
            # calculate the next datetime in utc
147
            # todo: timedelta of year
148
            next_datetime_utc = datetime(year=current_datetime_utc.year + 2,
149
                                         month=1,
150
                                         day=1,
151
                                         hour=current_datetime_utc.hour,
152
                                         minute=current_datetime_utc.minute,
153
                                         second=current_datetime_utc.second,
154
                                         microsecond=current_datetime_utc.microsecond,
155
                                         tzinfo=current_datetime_utc.tzinfo) - timedelta(days=1)
156
            subtotal = Decimal(0.0)
157
            for row in rows_hourly:
158
                if current_datetime_utc <= row[0] < next_datetime_utc:
159
                    subtotal += row[1]
160
161
            result_rows_yearly.append((current_datetime_utc, subtotal))
162
            current_datetime_utc = next_datetime_utc
163
        return result_rows_yearly
164
165
166
########################################################################################################################
167
# Get tariffs by energy category
168
########################################################################################################################
169 View Code Duplication
def get_energy_category_tariffs(cost_center_id, energy_category_id, start_datetime_utc, end_datetime_utc):
0 ignored issues
show
Duplication introduced by
This code seems to be duplicated in your project.
Loading history...
170
    # todo: validate parameters
171
    if cost_center_id is None:
172
        return dict()
173
174
    start_datetime_utc = start_datetime_utc.replace(tzinfo=None)
175
    end_datetime_utc = end_datetime_utc.replace(tzinfo=None)
176
177
    # get timezone offset in minutes, this value will be returned to client
178
    timezone_offset = int(config.utc_offset[1:3]) * 60 + int(config.utc_offset[4:6])
179
    if config.utc_offset[0] == '-':
180
        timezone_offset = -timezone_offset
181
182
    tariff_dict = collections.OrderedDict()
183
184
    cnx = None
185
    cursor = None
186
    try:
187
        cnx = mysql.connector.connect(**config.myems_system_db)
188
        cursor = cnx.cursor()
189
        query_tariffs = (" SELECT t.id, t.valid_from_datetime_utc, t.valid_through_datetime_utc "
190
                         " FROM tbl_tariffs t, tbl_cost_centers_tariffs cct "
191
                         " WHERE t.energy_category_id = %s AND "
192
                         "       t.id = cct.tariff_id AND "
193
                         "       cct.cost_center_id = %s AND "
194
                         "       t.valid_through_datetime_utc >= %s AND "
195
                         "       t.valid_from_datetime_utc <= %s "
196
                         " ORDER BY t.valid_from_datetime_utc ")
197
        cursor.execute(query_tariffs, (energy_category_id, cost_center_id, start_datetime_utc, end_datetime_utc,))
198
        rows_tariffs = cursor.fetchall()
199
    except Exception as e:
200
        print(str(e))
201
        if cnx:
202
            cnx.disconnect()
203
        if cursor:
204
            cursor.close()
205
        return dict()
206
207
    if rows_tariffs is None or len(rows_tariffs) == 0:
208
        if cursor:
209
            cursor.close()
210
        if cnx:
211
            cnx.disconnect()
212
        return dict()
213
214
    for row in rows_tariffs:
215
        tariff_dict[row[0]] = {'valid_from_datetime_utc': row[1],
216
                               'valid_through_datetime_utc': row[2],
217
                               'rates': list()}
218
219
    try:
220
        query_timeofuse_tariffs = (" SELECT tariff_id, start_time_of_day, end_time_of_day, price "
221
                                   " FROM tbl_tariffs_timeofuses "
222
                                   " WHERE tariff_id IN ( " + ', '.join(map(str, tariff_dict.keys())) + ")"
223
                                   " ORDER BY tariff_id, start_time_of_day ")
224
        cursor.execute(query_timeofuse_tariffs, )
225
        rows_timeofuse_tariffs = cursor.fetchall()
226
    except Exception as e:
227
        print(str(e))
228
        if cnx:
229
            cnx.disconnect()
230
        if cursor:
231
            cursor.close()
232
        return dict()
233
234
    if cursor:
235
        cursor.close()
236
    if cnx:
237
        cnx.disconnect()
238
239
    if rows_timeofuse_tariffs is None or len(rows_timeofuse_tariffs) == 0:
240
        return dict()
241
242
    for row in rows_timeofuse_tariffs:
243
        tariff_dict[row[0]]['rates'].append({'start_time_of_day': row[1],
244
                                             'end_time_of_day': row[2],
245
                                             'price': row[3]})
246
247
    result = dict()
248
    for tariff_id, tariff_value in tariff_dict.items():
249
        current_datetime_utc = tariff_value['valid_from_datetime_utc']
250
        while current_datetime_utc < tariff_value['valid_through_datetime_utc']:
251
            for rate in tariff_value['rates']:
252
                current_datetime_local = current_datetime_utc + timedelta(minutes=timezone_offset)
253
                seconds_since_midnight = (current_datetime_local -
254
                                          current_datetime_local.replace(hour=0,
255
                                                                         second=0,
256
                                                                         microsecond=0,
257
                                                                         tzinfo=None)).total_seconds()
258
                if rate['start_time_of_day'].total_seconds() <= \
259
                        seconds_since_midnight < rate['end_time_of_day'].total_seconds():
260
                    result[current_datetime_utc] = rate['price']
261
                    break
262
263
            # start from the next time slot
264
            current_datetime_utc += timedelta(minutes=config.minutes_to_count)
265
266
    return {k: v for k, v in result.items() if start_datetime_utc <= k <= end_datetime_utc}
267
268
269
########################################################################################################################
270
# Get peak types of tariff by energy category
271
# peak types: toppeak, onpeak, midpeak, offpeak
272
########################################################################################################################
273 View Code Duplication
def get_energy_category_peak_types(cost_center_id, energy_category_id, start_datetime_utc, end_datetime_utc):
0 ignored issues
show
Duplication introduced by
This code seems to be duplicated in your project.
Loading history...
274
    # todo: validate parameters
275
    if cost_center_id is None:
276
        return dict()
277
278
    start_datetime_utc = start_datetime_utc.replace(tzinfo=None)
279
    end_datetime_utc = end_datetime_utc.replace(tzinfo=None)
280
281
    # get timezone offset in minutes, this value will be returned to client
282
    timezone_offset = int(config.utc_offset[1:3]) * 60 + int(config.utc_offset[4:6])
283
    if config.utc_offset[0] == '-':
284
        timezone_offset = -timezone_offset
285
286
    tariff_dict = collections.OrderedDict()
287
288
    cnx = None
289
    cursor = None
290
    try:
291
        cnx = mysql.connector.connect(**config.myems_system_db)
292
        cursor = cnx.cursor()
293
        query_tariffs = (" SELECT t.id, t.valid_from_datetime_utc, t.valid_through_datetime_utc "
294
                         " FROM tbl_tariffs t, tbl_cost_centers_tariffs cct "
295
                         " WHERE t.energy_category_id = %s AND "
296
                         "       t.id = cct.tariff_id AND "
297
                         "       cct.cost_center_id = %s AND "
298
                         "       t.valid_through_datetime_utc >= %s AND "
299
                         "       t.valid_from_datetime_utc <= %s "
300
                         " ORDER BY t.valid_from_datetime_utc ")
301
        cursor.execute(query_tariffs, (energy_category_id, cost_center_id, start_datetime_utc, end_datetime_utc,))
302
        rows_tariffs = cursor.fetchall()
303
    except Exception as e:
304
        print(str(e))
305
        if cnx:
306
            cnx.disconnect()
307
        if cursor:
308
            cursor.close()
309
        return dict()
310
311
    if rows_tariffs is None or len(rows_tariffs) == 0:
312
        if cursor:
313
            cursor.close()
314
        if cnx:
315
            cnx.disconnect()
316
        return dict()
317
318
    for row in rows_tariffs:
319
        tariff_dict[row[0]] = {'valid_from_datetime_utc': row[1],
320
                               'valid_through_datetime_utc': row[2],
321
                               'rates': list()}
322
323
    try:
324
        query_timeofuse_tariffs = (" SELECT tariff_id, start_time_of_day, end_time_of_day, peak_type "
325
                                   " FROM tbl_tariffs_timeofuses "
326
                                   " WHERE tariff_id IN ( " + ', '.join(map(str, tariff_dict.keys())) + ")"
327
                                   " ORDER BY tariff_id, start_time_of_day ")
328
        cursor.execute(query_timeofuse_tariffs, )
329
        rows_timeofuse_tariffs = cursor.fetchall()
330
    except Exception as e:
331
        print(str(e))
332
        if cnx:
333
            cnx.disconnect()
334
        if cursor:
335
            cursor.close()
336
        return dict()
337
338
    if cursor:
339
        cursor.close()
340
    if cnx:
341
        cnx.disconnect()
342
343
    if rows_timeofuse_tariffs is None or len(rows_timeofuse_tariffs) == 0:
344
        return dict()
345
346
    for row in rows_timeofuse_tariffs:
347
        tariff_dict[row[0]]['rates'].append({'start_time_of_day': row[1],
348
                                             'end_time_of_day': row[2],
349
                                             'peak_type': row[3]})
350
351
    result = dict()
352
    for tariff_id, tariff_value in tariff_dict.items():
353
        current_datetime_utc = tariff_value['valid_from_datetime_utc']
354
        while current_datetime_utc < tariff_value['valid_through_datetime_utc']:
355
            for rate in tariff_value['rates']:
356
                current_datetime_local = current_datetime_utc + timedelta(minutes=timezone_offset)
357
                seconds_since_midnight = (current_datetime_local -
358
                                          current_datetime_local.replace(hour=0,
359
                                                                         second=0,
360
                                                                         microsecond=0,
361
                                                                         tzinfo=None)).total_seconds()
362
                if rate['start_time_of_day'].total_seconds() <= \
363
                        seconds_since_midnight < rate['end_time_of_day'].total_seconds():
364
                    result[current_datetime_utc] = rate['peak_type']
365
                    break
366
367
            # start from the next time slot
368
            current_datetime_utc += timedelta(minutes=config.minutes_to_count)
369
370
    return {k: v for k, v in result.items() if start_datetime_utc <= k <= end_datetime_utc}
371
372
373
########################################################################################################################
374
# Averaging calculator of hourly data by period
375
#   rows_hourly: list of (start_datetime_utc, actual_value), should belong to one energy_category_id
376
#   start_datetime_utc: start datetime in utc
377
#   end_datetime_utc: end datetime in utc
378
#   period_type: one of the following period types, 'hourly', 'daily', 'monthly' and 'yearly'
379
# Returns: periodically data of average and maximum
380
# Note: this procedure doesn't work with multiple energy categories
381
########################################################################################################################
382
def averaging_hourly_data_by_period(rows_hourly, start_datetime_utc, end_datetime_utc, period_type):
383
    # todo: validate parameters
384
    start_datetime_utc = start_datetime_utc.replace(tzinfo=None)
385
    end_datetime_utc = end_datetime_utc.replace(tzinfo=None)
386
387
    if period_type == "hourly":
388
        result_rows_hourly = list()
389
        # todo: add config.working_day_start_time_local
390
        # todo: add config.minutes_to_count
391
        total = Decimal(0.0)
392
        maximum = None
393
        counter = 0
394
        current_datetime_utc = start_datetime_utc.replace(minute=0, second=0, microsecond=0, tzinfo=None)
395 View Code Duplication
        while current_datetime_utc <= end_datetime_utc:
0 ignored issues
show
Duplication introduced by
This code seems to be duplicated in your project.
Loading history...
396
            sub_total = Decimal(0.0)
397
            sub_maximum = None
398
            sub_counter = 0
399
            for row in rows_hourly:
400
                if current_datetime_utc <= row[0] < current_datetime_utc + \
401
                        timedelta(minutes=config.minutes_to_count):
402
                    sub_total += row[1]
403
                    if sub_maximum is None:
404
                        sub_maximum = row[1]
405
                    elif sub_maximum < row[1]:
406
                        sub_maximum = row[1]
407
                    sub_counter += 1
408
409
            sub_average = (sub_total / sub_counter) if sub_counter > 0 else None
410
            result_rows_hourly.append((current_datetime_utc, sub_average, sub_maximum))
411
412
            total += sub_total
413
            counter += sub_counter
414
            if sub_maximum is None:
415
                pass
416
            elif maximum is None:
417
                maximum = sub_maximum
418
            elif maximum < sub_maximum:
419
                maximum = sub_maximum
420
421
            current_datetime_utc += timedelta(minutes=config.minutes_to_count)
422
423
        average = total / counter if counter > 0 else None
424
        return result_rows_hourly, average, maximum
425
426
    elif period_type == "daily":
427
        result_rows_daily = list()
428
        # todo: add config.working_day_start_time_local
429
        # todo: add config.minutes_to_count
430
        total = Decimal(0.0)
431
        maximum = None
432
        counter = 0
433
        # calculate the start datetime in utc of the first day in local
434
        start_datetime_local = start_datetime_utc + timedelta(hours=int(config.utc_offset[1:3]))
435
        current_datetime_utc = start_datetime_local.replace(hour=0) - timedelta(hours=int(config.utc_offset[1:3]))
436 View Code Duplication
        while current_datetime_utc <= end_datetime_utc:
0 ignored issues
show
Duplication introduced by
This code seems to be duplicated in your project.
Loading history...
437
            sub_total = Decimal(0.0)
438
            sub_maximum = None
439
            sub_counter = 0
440
            for row in rows_hourly:
441
                if current_datetime_utc <= row[0] < current_datetime_utc + timedelta(days=1):
442
                    sub_total += row[1]
443
                    if sub_maximum is None:
444
                        sub_maximum = row[1]
445
                    elif sub_maximum < row[1]:
446
                        sub_maximum = row[1]
447
                    sub_counter += 1
448
449
            sub_average = (sub_total / sub_counter) if sub_counter > 0 else None
450
            result_rows_daily.append((current_datetime_utc, sub_average, sub_maximum))
451
            total += sub_total
452
            counter += sub_counter
453
            if sub_maximum is None:
454
                pass
455
            elif maximum is None:
456
                maximum = sub_maximum
457
            elif maximum < sub_maximum:
458
                maximum = sub_maximum
459
            current_datetime_utc += timedelta(days=1)
460
461
        average = total / counter if counter > 0 else None
462
        return result_rows_daily, average, maximum
463
464
    elif period_type == "monthly":
465
        result_rows_monthly = list()
466
        # todo: add config.working_day_start_time_local
467
        # todo: add config.minutes_to_count
468
        total = Decimal(0.0)
469
        maximum = None
470
        counter = 0
471
        # calculate the start datetime in utc of the first day in the first month in local
472
        start_datetime_local = start_datetime_utc + timedelta(hours=int(config.utc_offset[1:3]))
473
        current_datetime_utc = \
474
            start_datetime_local.replace(day=1, hour=0) - timedelta(hours=int(config.utc_offset[1:3]))
475
476
        while current_datetime_utc <= end_datetime_utc:
477
            # calculate the next datetime in utc
478 View Code Duplication
            if current_datetime_utc.month == 1:
0 ignored issues
show
Duplication introduced by
This code seems to be duplicated in your project.
Loading history...
479
                temp_day = 28
480
                ny = current_datetime_utc.year
481
                if (ny % 100 != 0 and ny % 4 == 0) or (ny % 100 == 0 and ny % 400 == 0):
482
                    temp_day = 29
483
484
                next_datetime_utc = datetime(year=current_datetime_utc.year,
485
                                             month=current_datetime_utc.month + 1,
486
                                             day=temp_day,
487
                                             hour=current_datetime_utc.hour,
488
                                             minute=current_datetime_utc.minute,
489
                                             second=0,
490
                                             microsecond=0,
491
                                             tzinfo=None)
492
            elif current_datetime_utc.month == 2:
493
                next_datetime_utc = datetime(year=current_datetime_utc.year,
494
                                             month=current_datetime_utc.month + 1,
495
                                             day=31,
496
                                             hour=current_datetime_utc.hour,
497
                                             minute=current_datetime_utc.minute,
498
                                             second=0,
499
                                             microsecond=0,
500
                                             tzinfo=None)
501
            elif current_datetime_utc.month in [3, 5, 8, 10]:
502
                next_datetime_utc = datetime(year=current_datetime_utc.year,
503
                                             month=current_datetime_utc.month + 1,
504
                                             day=30,
505
                                             hour=current_datetime_utc.hour,
506
                                             minute=current_datetime_utc.minute,
507
                                             second=0,
508
                                             microsecond=0,
509
                                             tzinfo=None)
510
            elif current_datetime_utc.month == 7:
511
                next_datetime_utc = datetime(year=current_datetime_utc.year,
512
                                             month=current_datetime_utc.month + 1,
513
                                             day=31,
514
                                             hour=current_datetime_utc.hour,
515
                                             minute=current_datetime_utc.minute,
516
                                             second=0,
517
                                             microsecond=0,
518
                                             tzinfo=None)
519
            elif current_datetime_utc.month in [4, 6, 9, 11]:
520
                next_datetime_utc = datetime(year=current_datetime_utc.year,
521
                                             month=current_datetime_utc.month + 1,
522
                                             day=31,
523
                                             hour=current_datetime_utc.hour,
524
                                             minute=current_datetime_utc.minute,
525
                                             second=0,
526
                                             microsecond=0,
527
                                             tzinfo=None)
528
            elif current_datetime_utc.month == 12:
529
                next_datetime_utc = datetime(year=current_datetime_utc.year + 1,
530
                                             month=1,
531
                                             day=31,
532
                                             hour=current_datetime_utc.hour,
533
                                             minute=current_datetime_utc.minute,
534
                                             second=0,
535
                                             microsecond=0,
536
                                             tzinfo=None)
537
538
            sub_total = Decimal(0.0)
539
            sub_maximum = None
540
            sub_counter = 0
541
            for row in rows_hourly:
542
                if current_datetime_utc <= row[0] < next_datetime_utc:
0 ignored issues
show
introduced by
The variable next_datetime_utc does not seem to be defined for all execution paths.
Loading history...
543
                    sub_total += row[1]
544
                    if sub_maximum is None:
545
                        sub_maximum = row[1]
546
                    elif sub_maximum < row[1]:
547
                        sub_maximum = row[1]
548
                    sub_counter += 1
549
550
            sub_average = (sub_total / sub_counter) if sub_counter > 0 else None
551
            result_rows_monthly.append((current_datetime_utc, sub_average, sub_maximum))
552
            total += sub_total
553
            counter += sub_counter
554
            if sub_maximum is None:
555
                pass
556
            elif maximum is None:
557
                maximum = sub_maximum
558
            elif maximum < sub_maximum:
559
                maximum = sub_maximum
560
            current_datetime_utc = next_datetime_utc
561
562
        average = total / counter if counter > 0 else None
563
        return result_rows_monthly, average, maximum
564
565
    elif period_type == "yearly":
566
        result_rows_yearly = list()
567
        # todo: add config.working_day_start_time_local
568
        # todo: add config.minutes_to_count
569
        total = Decimal(0.0)
570
        maximum = None
571
        counter = 0
572
        # calculate the start datetime in utc of the first day in the first month in local
573
        start_datetime_local = start_datetime_utc + timedelta(hours=int(config.utc_offset[1:3]))
574
        current_datetime_utc = start_datetime_local.replace(month=1, day=1, hour=0) - timedelta(
575
            hours=int(config.utc_offset[1:3]))
576
577
        while current_datetime_utc <= end_datetime_utc:
578
            # calculate the next datetime in utc
579
            # todo: timedelta of year
580
            next_datetime_utc = datetime(year=current_datetime_utc.year + 2,
581
                                         month=1,
582
                                         day=1,
583
                                         hour=current_datetime_utc.hour,
584
                                         minute=current_datetime_utc.minute,
585
                                         second=current_datetime_utc.second,
586
                                         microsecond=current_datetime_utc.microsecond,
587
                                         tzinfo=current_datetime_utc.tzinfo) - timedelta(days=1)
588
            sub_total = Decimal(0.0)
589
            sub_maximum = None
590
            sub_counter = 0
591
            for row in rows_hourly:
592
                if current_datetime_utc <= row[0] < next_datetime_utc:
593
                    sub_total += row[1]
594
                    if sub_maximum is None:
595
                        sub_maximum = row[1]
596
                    elif sub_maximum < row[1]:
597
                        sub_maximum = row[1]
598
                    sub_counter += 1
599
600
            sub_average = (sub_total / sub_counter) if sub_counter > 0 else None
601
            result_rows_yearly.append((current_datetime_utc, sub_average, sub_maximum))
602
            total += sub_total
603
            counter += sub_counter
604
            if sub_maximum is None:
605
                pass
606
            elif maximum is None:
607
                maximum = sub_maximum
608
            elif maximum < sub_maximum:
609
                maximum = sub_maximum
610
            current_datetime_utc = next_datetime_utc
611
612
        average = total / counter if counter > 0 else None
613
        return result_rows_yearly, average, maximum
614
615
616
########################################################################################################################
617
# Statistics calculator of hourly data by period
618
#   rows_hourly: list of (start_datetime_utc, actual_value), should belong to one energy_category_id
619
#   start_datetime_utc: start datetime in utc
620
#   end_datetime_utc: end datetime in utc
621
#   period_type: one of the following period types, 'hourly', 'daily', 'monthly' and 'yearly'
622
# Returns: periodically data of values and statistics of mean, median, minimum, maximum, stdev and variance
623
# Note: this procedure doesn't work with multiple energy categories
624
########################################################################################################################
625
def statistics_hourly_data_by_period(rows_hourly, start_datetime_utc, end_datetime_utc, period_type):
626
    # todo: validate parameters
627
    start_datetime_utc = start_datetime_utc.replace(tzinfo=None)
628
    end_datetime_utc = end_datetime_utc.replace(tzinfo=None)
629
630
    if period_type == "hourly":
631
        result_rows_hourly = list()
632
        sample_data = list()
633
        # todo: add config.working_day_start_time_local
634
        # todo: add config.minutes_to_count
635
        counter = 0
636
        mean = None
637
        median = None
638
        minimum = None
639
        maximum = None
640
        stdev = None
641
        variance = None
642
        current_datetime_utc = start_datetime_utc.replace(minute=0, second=0, microsecond=0, tzinfo=None)
643 View Code Duplication
        while current_datetime_utc <= end_datetime_utc:
0 ignored issues
show
Duplication introduced by
This code seems to be duplicated in your project.
Loading history...
644
            sub_total = Decimal(0.0)
645
            for row in rows_hourly:
646
                if current_datetime_utc <= row[0] < current_datetime_utc + \
647
                        timedelta(minutes=config.minutes_to_count):
648
                    sub_total += row[1]
649
650
            result_rows_hourly.append((current_datetime_utc, sub_total))
651
            sample_data.append(sub_total)
652
653
            counter += 1
654
            if minimum is None:
655
                minimum = sub_total
656
            elif minimum > sub_total:
657
                minimum = sub_total
658
659
            if maximum is None:
660
                maximum = sub_total
661
            elif maximum < sub_total:
662
                maximum = sub_total
663
664
            current_datetime_utc += timedelta(minutes=config.minutes_to_count)
665
666
        if len(sample_data) > 1:
667
            mean = statistics.mean(sample_data)
668
            median = statistics.median(sample_data)
669
            stdev = statistics.stdev(sample_data)
670
            variance = statistics.variance(sample_data)
671
672
        return result_rows_hourly, mean, median, minimum, maximum, stdev, variance
673
674
    elif period_type == "daily":
675
        result_rows_daily = list()
676
        sample_data = list()
677
        # todo: add config.working_day_start_time_local
678
        # todo: add config.minutes_to_count
679
        counter = 0
680
        mean = None
681
        median = None
682
        minimum = None
683
        maximum = None
684
        stdev = None
685
        variance = None
686
        # calculate the start datetime in utc of the first day in local
687
        start_datetime_local = start_datetime_utc + timedelta(hours=int(config.utc_offset[1:3]))
688
        current_datetime_utc = start_datetime_local.replace(hour=0) - timedelta(hours=int(config.utc_offset[1:3]))
689 View Code Duplication
        while current_datetime_utc <= end_datetime_utc:
0 ignored issues
show
Duplication introduced by
This code seems to be duplicated in your project.
Loading history...
690
            sub_total = Decimal(0.0)
691
            for row in rows_hourly:
692
                if current_datetime_utc <= row[0] < current_datetime_utc + timedelta(days=1):
693
                    sub_total += row[1]
694
695
            result_rows_daily.append((current_datetime_utc, sub_total))
696
            sample_data.append(sub_total)
697
698
            counter += 1
699
            if minimum is None:
700
                minimum = sub_total
701
            elif minimum > sub_total:
702
                minimum = sub_total
703
704
            if maximum is None:
705
                maximum = sub_total
706
            elif maximum < sub_total:
707
                maximum = sub_total
708
            current_datetime_utc += timedelta(days=1)
709
710
        if len(sample_data) > 1:
711
            mean = statistics.mean(sample_data)
712
            median = statistics.median(sample_data)
713
            stdev = statistics.stdev(sample_data)
714
            variance = statistics.variance(sample_data)
715
716
        return result_rows_daily, mean, median, minimum, maximum, stdev, variance
717
718
    elif period_type == "monthly":
719
        result_rows_monthly = list()
720
        sample_data = list()
721
        # todo: add config.working_day_start_time_local
722
        # todo: add config.minutes_to_count
723
        counter = 0
724
        mean = None
725
        median = None
726
        minimum = None
727
        maximum = None
728
        stdev = None
729
        variance = None
730
        # calculate the start datetime in utc of the first day in the first month in local
731
        start_datetime_local = start_datetime_utc + timedelta(hours=int(config.utc_offset[1:3]))
732
        current_datetime_utc = \
733
            start_datetime_local.replace(day=1, hour=0) - timedelta(hours=int(config.utc_offset[1:3]))
734
735
        while current_datetime_utc <= end_datetime_utc:
736
            # calculate the next datetime in utc
737 View Code Duplication
            if current_datetime_utc.month == 1:
0 ignored issues
show
Duplication introduced by
This code seems to be duplicated in your project.
Loading history...
738
                temp_day = 28
739
                ny = current_datetime_utc.year
740
                if (ny % 100 != 0 and ny % 4 == 0) or (ny % 100 == 0 and ny % 400 == 0):
741
                    temp_day = 29
742
743
                next_datetime_utc = datetime(year=current_datetime_utc.year,
744
                                             month=current_datetime_utc.month + 1,
745
                                             day=temp_day,
746
                                             hour=current_datetime_utc.hour,
747
                                             minute=current_datetime_utc.minute,
748
                                             second=0,
749
                                             microsecond=0,
750
                                             tzinfo=None)
751
            elif current_datetime_utc.month == 2:
752
                next_datetime_utc = datetime(year=current_datetime_utc.year,
753
                                             month=current_datetime_utc.month + 1,
754
                                             day=31,
755
                                             hour=current_datetime_utc.hour,
756
                                             minute=current_datetime_utc.minute,
757
                                             second=0,
758
                                             microsecond=0,
759
                                             tzinfo=None)
760
            elif current_datetime_utc.month in [3, 5, 8, 10]:
761
                next_datetime_utc = datetime(year=current_datetime_utc.year,
762
                                             month=current_datetime_utc.month + 1,
763
                                             day=30,
764
                                             hour=current_datetime_utc.hour,
765
                                             minute=current_datetime_utc.minute,
766
                                             second=0,
767
                                             microsecond=0,
768
                                             tzinfo=None)
769
            elif current_datetime_utc.month == 7:
770
                next_datetime_utc = datetime(year=current_datetime_utc.year,
771
                                             month=current_datetime_utc.month + 1,
772
                                             day=31,
773
                                             hour=current_datetime_utc.hour,
774
                                             minute=current_datetime_utc.minute,
775
                                             second=0,
776
                                             microsecond=0,
777
                                             tzinfo=None)
778
            elif current_datetime_utc.month in [4, 6, 9, 11]:
779
                next_datetime_utc = datetime(year=current_datetime_utc.year,
780
                                             month=current_datetime_utc.month + 1,
781
                                             day=31,
782
                                             hour=current_datetime_utc.hour,
783
                                             minute=current_datetime_utc.minute,
784
                                             second=0,
785
                                             microsecond=0,
786
                                             tzinfo=None)
787
            elif current_datetime_utc.month == 12:
788
                next_datetime_utc = datetime(year=current_datetime_utc.year + 1,
789
                                             month=1,
790
                                             day=31,
791
                                             hour=current_datetime_utc.hour,
792
                                             minute=current_datetime_utc.minute,
793
                                             second=0,
794
                                             microsecond=0,
795
                                             tzinfo=None)
796
797
            sub_total = Decimal(0.0)
798
            for row in rows_hourly:
799
                if current_datetime_utc <= row[0] < next_datetime_utc:
0 ignored issues
show
introduced by
The variable next_datetime_utc does not seem to be defined for all execution paths.
Loading history...
800
                    sub_total += row[1]
801
802
            result_rows_monthly.append((current_datetime_utc, sub_total))
803
            sample_data.append(sub_total)
804
805
            counter += 1
806
            if minimum is None:
807
                minimum = sub_total
808
            elif minimum > sub_total:
809
                minimum = sub_total
810
811
            if maximum is None:
812
                maximum = sub_total
813
            elif maximum < sub_total:
814
                maximum = sub_total
815
            current_datetime_utc = next_datetime_utc
816
817
        if len(sample_data) > 1:
818
            mean = statistics.mean(sample_data)
819
            median = statistics.median(sample_data)
820
            stdev = statistics.stdev(sample_data)
821
            variance = statistics.variance(sample_data)
822
823
        return result_rows_monthly, mean, median, minimum, maximum, stdev, variance
824
825
    elif period_type == "yearly":
826
        result_rows_yearly = list()
827
        sample_data = list()
828
        # todo: add config.working_day_start_time_local
829
        # todo: add config.minutes_to_count
830
        mean = None
831
        median = None
832
        minimum = None
833
        maximum = None
834
        stdev = None
835
        variance = None
836
        # calculate the start datetime in utc of the first day in the first month in local
837
        start_datetime_local = start_datetime_utc + timedelta(hours=int(config.utc_offset[1:3]))
838
        current_datetime_utc = start_datetime_local.replace(month=1, day=1, hour=0) - timedelta(
839
            hours=int(config.utc_offset[1:3]))
840
841
        while current_datetime_utc <= end_datetime_utc:
842
            # calculate the next datetime in utc
843
            # todo: timedelta of year
844
            next_datetime_utc = datetime(year=current_datetime_utc.year + 2,
845
                                         month=1,
846
                                         day=1,
847
                                         hour=current_datetime_utc.hour,
848
                                         minute=current_datetime_utc.minute,
849
                                         second=current_datetime_utc.second,
850
                                         microsecond=current_datetime_utc.microsecond,
851
                                         tzinfo=current_datetime_utc.tzinfo) - timedelta(days=1)
852
            sub_total = Decimal(0.0)
853
            for row in rows_hourly:
854
                if current_datetime_utc <= row[0] < next_datetime_utc:
855
                    sub_total += row[1]
856
857
            result_rows_yearly.append((current_datetime_utc, sub_total))
858
            sample_data.append(sub_total)
859
860
            if minimum is None:
861
                minimum = sub_total
862
            elif minimum > sub_total:
863
                minimum = sub_total
864
            if maximum is None:
865
                maximum = sub_total
866
            elif maximum < sub_total:
867
                maximum = sub_total
868
869
            current_datetime_utc = next_datetime_utc
870
871
        if len(sample_data) > 1:
872
            mean = statistics.mean(sample_data)
873
            median = statistics.median(sample_data)
874
            stdev = statistics.stdev(sample_data)
875
            variance = statistics.variance(sample_data)
876
877
        return result_rows_yearly, mean, median, minimum, maximum, stdev, variance
878