virtual_meter_billing.main()   F
last analyzed

Complexity

Conditions 53

Size

Total Lines 281
Code Lines 172

Duplication

Lines 244
Ratio 86.83 %

Importance

Changes 0
Metric Value
eloc 172
dl 244
loc 281
rs 0
c 0
b 0
f 0
cc 53
nop 1

How to fix   Long Method    Complexity   

Long Method

Small methods make your code easier to understand, in particular if combined with a good name. Besides, if your method is small, finding a good name is usually much easier.

For example, if you find yourself adding comments to a method's body, this is usually a good sign to extract the commented part to a new method, and use the comment as a starting point when coming up with a good name for this new method.

Commonly applied refactorings include:

Complexity

Complex classes like virtual_meter_billing.main() often do a lot of different things. To break such a class down, we need to identify a cohesive component within that class. A common approach to find such a component is to look for fields/methods that share the same prefixes, or suffixes.

Once you have determined the fields that belong together, you can apply the Extract Class refactoring. If the component makes sense as a sub-class, Extract Subclass is also a candidate, and is often faster.

1
"""
2
MyEMS Aggregation Service - Virtual Meter Billing Module
3
4
This module handles billing calculations for virtual meters based on energy consumption
5
and tariff structures. Virtual meters are calculated meters that derive their values from
6
mathematical expressions involving other meters, enabling complex energy calculations.
7
8
The virtual meter billing process performs the following functions:
9
1. Retrieves all virtual meters from the system database
10
2. For each virtual meter, determines the latest processed billing data
11
3. Fetches calculated energy consumption data since the last processed time
12
4. Retrieves applicable tariffs for the meter's energy category and cost center
13
5. Calculates billing costs by multiplying energy consumption with tariff rates
14
6. Stores billing data in the billing database
15
16
Key features:
17
- Handles time-of-use pricing with different rates for different time periods
18
- Processes billing calculations for virtual meters with calculated energy values
19
- Supports incremental processing to avoid recalculating existing data
20
- Maintains data integrity through comprehensive error handling
21
"""
22
23
import time
24
from datetime import datetime, timedelta
25
from decimal import Decimal
26
27
import mysql.connector
28
29
import config
30
import tariff
31
32
33
########################################################################################################################
34
# Virtual Meter Billing Calculation Procedures:
35
# Step 1: Get all virtual meters from system database
36
# For each virtual meter in list:
37
#   Step 2: Get the latest start_datetime_utc from billing database
38
#   Step 3: Get all energy data since the latest start_datetime_utc
39
#   Step 4: Get tariffs for the meter's energy category and cost center
40
#   Step 5: Calculate billing by multiplying energy consumption with tariff rates
41
#   Step 6: Save billing data to billing database
42
########################################################################################################################
43
44
45 View Code Duplication
def main(logger):
0 ignored issues
show
Duplication introduced by
This code seems to be duplicated in your project.
Loading history...
46
    """
47
    Main function for virtual meter billing calculation.
48
49
    This function runs continuously, processing billing calculations for all virtual meters.
50
    It retrieves calculated energy consumption data from mathematical expressions, applies tariff rates,
51
    and calculates billing costs for each virtual meter based on their energy category and cost center.
52
53
    Args:
54
        logger: Logger instance for recording billing activities and errors
55
    """
56
    while True:
57
        # The outermost while loop to handle database connection errors and retry
58
        ################################################################################################################
59
        # Step 1: Get all virtual meters from system database
60
        ################################################################################################################
61
        cnx_system_db = None
62
        cursor_system_db = None
63
64
        # Connect to system database to retrieve virtual meter information
65
        try:
66
            cnx_system_db = mysql.connector.connect(**config.myems_system_db)
67
            cursor_system_db = cnx_system_db.cursor()
68
        except Exception as e:
69
            logger.error("Error in step 1.1 of virtual_meter_billing " + str(e))
70
            if cursor_system_db:
71
                cursor_system_db.close()
72
            if cnx_system_db:
73
                cnx_system_db.close()
74
            # Sleep and continue the outermost while loop to retry connection
75
            time.sleep(60)
76
            continue
77
78
        print("Connected to MyEMS System Database")
79
80
        # Retrieve all virtual meters with their energy category and cost center information
81
        virtual_meter_list = list()
82
        try:
83
            cursor_system_db.execute(" SELECT id, name, energy_category_id, cost_center_id "
84
                                     " FROM tbl_virtual_meters "
85
                                     " ORDER BY id ")
86
            rows_virtual_meters = cursor_system_db.fetchall()
87
88
            # Check if virtual meters were found
89
            if rows_virtual_meters is None or len(rows_virtual_meters) == 0:
90
                print("Step 1.2: There isn't any virtual meters ")
91
                if cursor_system_db:
92
                    cursor_system_db.close()
93
                if cnx_system_db:
94
                    cnx_system_db.close()
95
                # Sleep and continue the outermost while loop
96
                time.sleep(60)
97
                continue
98
99
            # Build virtual meter list with configuration data
100
            for row in rows_virtual_meters:
101
                virtual_meter_list.append({"id": row[0],
102
                                           "name": row[1],
103
                                           "energy_category_id": row[2],
104
                                           "cost_center_id": row[3]})
105
106
        except Exception as e:
107
            logger.error("Error in step 1.2 of virtual_meter_billing " + str(e))
108
            if cursor_system_db:
109
                cursor_system_db.close()
110
            if cnx_system_db:
111
                cnx_system_db.close()
112
            # Sleep and continue the outermost while loop
113
            time.sleep(60)
114
            continue
115
116
        print("Step 1.2: Got all virtual meters from MyEMS System Database")
117
118
        # Connect to energy database to retrieve energy consumption data
119
        cnx_energy_db = None
120
        cursor_energy_db = None
121
        try:
122
            cnx_energy_db = mysql.connector.connect(**config.myems_energy_db)
123
            cursor_energy_db = cnx_energy_db.cursor()
124
        except Exception as e:
125
            logger.error("Error in step 1.3 of virtual_meter_billing " + str(e))
126
            if cursor_energy_db:
127
                cursor_energy_db.close()
128
            if cnx_energy_db:
129
                cnx_energy_db.close()
130
131
            if cursor_system_db:
132
                cursor_system_db.close()
133
            if cnx_system_db:
134
                cnx_system_db.close()
135
            # Sleep and continue the outermost while loop
136
            time.sleep(60)
137
            continue
138
139
        print("Connected to MyEMS Energy Database")
140
141
        # Connect to billing database to store calculated billing data
142
        cnx_billing_db = None
143
        cursor_billing_db = None
144
        try:
145
            cnx_billing_db = mysql.connector.connect(**config.myems_billing_db)
146
            cursor_billing_db = cnx_billing_db.cursor()
147
        except Exception as e:
148
            logger.error("Error in step 1.4 of virtual_meter_billing " + str(e))
149
            if cursor_billing_db:
150
                cursor_billing_db.close()
151
            if cnx_billing_db:
152
                cnx_billing_db.close()
153
154
            if cursor_energy_db:
155
                cursor_energy_db.close()
156
            if cnx_energy_db:
157
                cnx_energy_db.close()
158
159
            if cursor_system_db:
160
                cursor_system_db.close()
161
            if cnx_system_db:
162
                cnx_system_db.close()
163
            # Sleep and continue the outermost while loop
164
            time.sleep(60)
165
            continue
166
167
        print("Connected to MyEMS Billing Database")
168
169
        # Process each virtual meter for billing calculation
170
        for virtual_meter in virtual_meter_list:
171
172
            ############################################################################################################
173
            # Step 2: Get the latest start_datetime_utc from billing database
174
            ############################################################################################################
175
            print("Step 2: get the latest start_datetime_utc from billing database for " + virtual_meter['name'])
176
            try:
177
                # Query for the latest processed billing data to determine where to continue
178
                cursor_billing_db.execute(" SELECT MAX(start_datetime_utc) "
179
                                          " FROM tbl_virtual_meter_hourly "
180
                                          " WHERE virtual_meter_id = %s ",
181
                                          (virtual_meter['id'], ))
182
                row_datetime = cursor_billing_db.fetchone()
183
184
                # Initialize start datetime from configuration
185
                start_datetime_utc = datetime.strptime(config.start_datetime_utc, '%Y-%m-%d %H:%M:%S')
186
                start_datetime_utc = start_datetime_utc.replace(minute=0, second=0, microsecond=0, tzinfo=None)
187
188
                # Update start datetime if existing billing data is found
189
                if row_datetime is not None and len(row_datetime) > 0 and isinstance(row_datetime[0], datetime):
190
                    # Replace second and microsecond with 0
191
                    # Note: Do not replace minute in case of calculating in half hourly
192
                    start_datetime_utc = row_datetime[0].replace(second=0, microsecond=0, tzinfo=None)
193
                    # Start from the next time slot
194
                    start_datetime_utc += timedelta(minutes=config.minutes_to_count)
195
196
                print("start_datetime_utc: " + start_datetime_utc.isoformat()[0:19])
197
            except Exception as e:
198
                logger.error("Error in step 2 of virtual_meter_billing " + str(e))
199
                # Break the for virtual_meter loop
200
                break
201
202
            ############################################################################################################
203
            # Step 3: Get all energy data since the latest start_datetime_utc
204
            ############################################################################################################
205
            print("Step 3: get all energy data since the latest start_datetime_utc")
206
            try:
207
                # Query for calculated energy consumption data from the energy database
208
                query = (" SELECT start_datetime_utc, actual_value "
209
                         " FROM tbl_virtual_meter_hourly "
210
                         " WHERE virtual_meter_id = %s AND start_datetime_utc >= %s "
211
                         " ORDER BY id ")
212
                cursor_energy_db.execute(query, (virtual_meter['id'], start_datetime_utc, ))
213
                rows_hourly = cursor_energy_db.fetchall()
214
215
                # Check if energy data is available
216
                if rows_hourly is None or len(rows_hourly) == 0:
217
                    print("Step 3: There isn't any energy input data to calculate. ")
218
                    # Continue the for virtual_meter loop
219
                    continue
220
221
                # Build energy consumption dictionary and determine end datetime
222
                energy_dict = dict()
223
                end_datetime_utc = start_datetime_utc
224
                for row_hourly in rows_hourly:
225
                    current_datetime_utc = row_hourly[0]
226
                    actual_value = row_hourly[1]
227
                    if energy_dict.get(current_datetime_utc) is None:
228
                        energy_dict[current_datetime_utc] = dict()
229
                    energy_dict[current_datetime_utc][virtual_meter['energy_category_id']] = actual_value
230
                    if current_datetime_utc > end_datetime_utc:
231
                        end_datetime_utc = current_datetime_utc
232
            except Exception as e:
233
                logger.error("Error in step 3 of virtual_meter_billing " + str(e))
234
                # Break the for virtual_meter loop
235
                break
236
237
            ############################################################################################################
238
            # Step 4: Get tariffs for the virtual meter's energy category and cost center
239
            ############################################################################################################
240
            print("Step 4: get tariffs")
241
            tariff_dict = dict()
242
            # Retrieve tariff information for the virtual meter's energy category and cost center
243
            tariff_dict[virtual_meter['energy_category_id']] = \
244
                tariff.get_energy_category_tariffs(virtual_meter['cost_center_id'],
245
                                                   virtual_meter['energy_category_id'],
246
                                                   start_datetime_utc,
247
                                                   end_datetime_utc)
248
249
            ############################################################################################################
250
            # Step 5: Calculate billing by multiplying energy consumption with tariff rates
251
            ############################################################################################################
252
            print("Step 5: calculate billing by multiplying energy with tariff")
253
            aggregated_values = list()
254
255
            # Calculate billing costs for each time slot
256
            if len(energy_dict) > 0:
257
                for current_datetime_utc in energy_dict.keys():
258
                    aggregated_value = dict()
259
                    aggregated_value['start_datetime_utc'] = current_datetime_utc
260
                    aggregated_value['actual_value'] = None
261
262
                    # Get tariff rate and energy consumption for current time slot
263
                    current_tariff = tariff_dict[virtual_meter['energy_category_id']].get(current_datetime_utc)
264
                    current_energy = energy_dict[current_datetime_utc].get(virtual_meter['energy_category_id'])
265
266
                    # Calculate billing cost if both tariff and energy data are available
267
                    if current_tariff is not None \
268
                            and isinstance(current_tariff, Decimal) \
269
                            and current_energy is not None \
270
                            and isinstance(current_energy, Decimal):
271
                        aggregated_value['actual_value'] = current_energy * current_tariff
272
                        aggregated_values.append(aggregated_value)
273
274
            ############################################################################################################
275
            # Step 6: Save billing data to billing database
276
            ############################################################################################################
277
            print("Step 6: save billing data to billing database")
278
279
            # Process calculated billing values in batches of 100 to avoid overwhelming the database
280
            while len(aggregated_values) > 0:
281
                insert_100 = aggregated_values[:100]  # Take first 100 items
282
                aggregated_values = aggregated_values[100:]  # Remove processed items
283
284
                try:
285
                    # Build INSERT statement for virtual meter billing data
286
                    add_values = (" INSERT INTO tbl_virtual_meter_hourly "
287
                                  "             (virtual_meter_id, "
288
                                  "              start_datetime_utc, "
289
                                  "              actual_value) "
290
                                  " VALUES  ")
291
292
                    # Add each billing value to the INSERT statement
293
                    for aggregated_value in insert_100:
294
                        if aggregated_value['actual_value'] is not None and \
295
                                isinstance(aggregated_value['actual_value'], Decimal):
296
                            add_values += " (" + str(virtual_meter['id']) + ","
297
                            add_values += "'" + aggregated_value['start_datetime_utc'].isoformat()[0:19] + "',"
298
                            add_values += str(aggregated_value['actual_value']) + "), "
299
300
                    # Trim ", " at the end of string and then execute
301
                    cursor_billing_db.execute(add_values[:-2])
302
                    cnx_billing_db.commit()
303
                except Exception as e:
304
                    logger.error("Error in step 6 of virtual_meter_billing " + str(e))
305
                    break
306
307
        # End of for virtual_meter loop - clean up database connections
308
        if cursor_system_db:
309
            cursor_system_db.close()
310
        if cnx_system_db:
311
            cnx_system_db.close()
312
313
        if cursor_energy_db:
314
            cursor_energy_db.close()
315
        if cnx_energy_db:
316
            cnx_energy_db.close()
317
318
        if cursor_billing_db:
319
            cursor_billing_db.close()
320
        if cnx_billing_db:
321
            cnx_billing_db.close()
322
323
        print("go to sleep 300 seconds...")
324
        time.sleep(300)  # Sleep for 5 minutes before next processing cycle
325
        print("wake from sleep, and continue to work...")
326
    # End of the outermost while loop
327