GitHub Access Token became invalid

It seems like the GitHub access token used for retrieving details about this repository from GitHub became invalid. This might prevent certain types of inspections from being run (in particular, everything related to pull requests).
Please ask an admin of your repository to re-new the access token on this website.
Passed
Push — master ( 0342e8...c99b2a )
by P.R.
02:59
created

Spawner.daemonize()   A

Complexity

Conditions 1

Size

Total Lines 6

Duplication

Lines 0
Ratio 0 %

Code Coverage

Tests 0
CRAP Score 2

Importance

Changes 0
Metric Value
cc 1
dl 0
loc 6
ccs 0
cts 2
cp 0
crap 2
rs 9.4285
c 0
b 0
f 0
1
"""
2
Enarksh
3
4
Copyright 2013-2016 Set Based IT Consultancy
5
6
Licence MIT
7
"""
8
import os
9
import signal
10
import sys
11
import select
12
import pwd
13
import zmq
14
import enarksh
15
from enarksh import SPAWNER_PULL_END_POINT, CONTROLLER_PULL_END_POINT, LOGGER_PULL_END_POINT
16
from enarksh.Spawner.JobHandler import JobHandler
17
18
19
class Spawner:
0 ignored issues
show
Coding Style introduced by
This class should have a docstring.

The coding style of this project requires that you add a docstring to this code element. Below, you find an example for methods:

class SomeClass:
    def some_method(self):
        """Do x and return foo."""

If you would like to know more about docstrings, we recommend to read PEP-257: Docstring Conventions.

Loading history...
20
    _instance = None
21
22
    # ------------------------------------------------------------------------------------------------------------------
23
    def __init__(self):
24
        Spawner._instance = self
25
26
        self._child_flag = False
27
        self._zmq_context = None
28
        self._zmq_pull_socket = None
29
        self._zmq_controller = None
30
        self._zmq_logger = None
31
32
        self._job_handlers = {}
33
34
    # ------------------------------------------------------------------------------------------------------------------
35
    def _zmq_init(self):
0 ignored issues
show
Coding Style introduced by
This method should have a docstring.

The coding style of this project requires that you add a docstring to this code element. Below, you find an example for methods:

class SomeClass:
    def some_method(self):
        """Do x and return foo."""

If you would like to know more about docstrings, we recommend to read PEP-257: Docstring Conventions.

Loading history...
36
        self._zmq_context = zmq.Context()
37
38
        # Create socket for asynchronous incoming messages.
39
        self._zmq_pull_socket = self._zmq_context.socket(zmq.PULL)
0 ignored issues
show
Bug introduced by
The Module zmq does not seem to have a member named PULL.

This check looks for calls to members that are non-existent. These calls will fail.

The member could have been renamed or removed.

Loading history...
40
        self._zmq_pull_socket.bind(SPAWNER_PULL_END_POINT)
41
42
        # Create socket for sending asynchronous messages to the controller.
43
        self._zmq_controller = self._zmq_context.socket(zmq.PUSH)
1 ignored issue
show
Bug introduced by
The Module zmq does not seem to have a member named PUSH.

This check looks for calls to members that are non-existent. These calls will fail.

The member could have been renamed or removed.

Loading history...
44
        self._zmq_controller.connect(CONTROLLER_PULL_END_POINT)
45
46
        # Create socket for sending asynchronous messages to the logger.
47
        self._zmq_logger = self._zmq_context.socket(zmq.PUSH)
1 ignored issue
show
Bug introduced by
The Module zmq does not seem to have a member named PUSH.

This check looks for calls to members that are non-existent. These calls will fail.

The member could have been renamed or removed.

Loading history...
48
        self._zmq_logger.connect(LOGGER_PULL_END_POINT)
49
50
    # ------------------------------------------------------------------------------------------------------------------
51
    @staticmethod
52
    def daemonize():
0 ignored issues
show
Coding Style introduced by
This method should have a docstring.

The coding style of this project requires that you add a docstring to this code element. Below, you find an example for methods:

class SomeClass:
    def some_method(self):
        """Do x and return foo."""

If you would like to know more about docstrings, we recommend to read PEP-257: Docstring Conventions.

Loading history...
53
        enarksh.daemonize(enarksh.HOME + '/var/lock/spawnerd.pid',
54
                          '/dev/null',
55
                          enarksh.HOME + '/var/log/spawnerd.log',
56
                          enarksh.HOME + '/var/log/spawnerd.log')
57
58
    # ------------------------------------------------------------------------------------------------------------------
59
    @staticmethod
60
    def sigchld_handler(signum, frame):
0 ignored issues
show
Unused Code introduced by
The argument frame seems to be unused.
Loading history...
Unused Code introduced by
The argument signum seems to be unused.
Loading history...
61
        """
62
        Static method for SIGCHLD. Set a flag that a child has exited.
63
        """
64
        Spawner._instance._child_flag = True
0 ignored issues
show
Coding Style Best Practice introduced by
It seems like _child_flag was declared protected and should not be accessed from this context.

Prefixing a member variable _ is usually regarded as the equivalent of declaring it with protected visibility that exists in other languages. Consequentially, such a member should only be accessed from the same class or a child class:

class MyParent:
    def __init__(self):
        self._x = 1;
        self.y = 2;

class MyChild(MyParent):
    def some_method(self):
        return self._x    # Ok, since accessed from a child class

class AnotherClass:
    def some_method(self, instance_of_my_child):
        return instance_of_my_child._x   # Would be flagged as AnotherClass is not
                                         # a child class of MyParent
Loading history...
65
66
    # ------------------------------------------------------------------------------------------------------------------
67
    @staticmethod
68
    def sighup_handler(signum, frame):
0 ignored issues
show
Unused Code introduced by
The argument frame seems to be unused.
Loading history...
Unused Code introduced by
The argument signum seems to be unused.
Loading history...
69
        """
70
        SIGHUP is send by the log rotate daemon. CLoses the current log file and create a new log file.
71
        """
72
        sys.stdout.flush()
73
        sys.stderr.flush()
74
75
        with open(enarksh.HOME + '/var/log/spawnerd.log', 'wb', 0) as f:
0 ignored issues
show
Coding Style Naming introduced by
The name f does not conform to the variable naming conventions ([a-z_][a-z0-9_]{1,60}$).

This check looks for invalid names for a range of different identifiers.

You can set regular expressions to which the identifiers must conform if the defaults do not match your requirements.

If your project includes a Pylint configuration file, the settings contained in that file take precedence.

To find out more about Pylint, please refer to their site.

Loading history...
76
            os.dup2(f.fileno(), sys.stdout.fileno())
77
78
        with open(enarksh.HOME + '/var/log/spawnerd.log', 'wb', 0) as f:
0 ignored issues
show
Coding Style Naming introduced by
The name f does not conform to the variable naming conventions ([a-z_][a-z0-9_]{1,60}$).

This check looks for invalid names for a range of different identifiers.

You can set regular expressions to which the identifiers must conform if the defaults do not match your requirements.

If your project includes a Pylint configuration file, the settings contained in that file take precedence.

To find out more about Pylint, please refer to their site.

Loading history...
79
            os.dup2(f.fileno(), sys.stderr.fileno())
80
81
    # ------------------------------------------------------------------------------------------------------------------
82
    def _install_signal_handlers(self):
83
        """
84
        Install signal handlers for SIGCHLD and SIGHUP.
85
        """
86
        # Install signal handler for child has exited.
87
        signal.signal(signal.SIGCHLD, self.sigchld_handler)
88
89
        # Install signal handler log rotate.
90
        signal.signal(signal.SIGHUP, self.sighup_handler)
91
92
        # Unfortunately, restart system calls doesnt work.
93
        # signal.siginterrupt(signal.SIGCHLD, False)
94
        # signal.siginterrupt(signal.SIGHUP, False)
95
96
    # ------------------------------------------------------------------------------------------------------------------
97
    @staticmethod
98
    def _set_unprivileged_user():
99
        """
100
        Set the real and effective user and group to an unprivileged user.
101
        """
102
        _, _, uid, gid, _, _, _ = pwd.getpwnam('enarksh')
103
104
        os.setresgid(gid, gid, 0)
105
        os.setresuid(uid, uid, 0)
106
107
    # ------------------------------------------------------------------------------------------------------------------
108
    def _startup(self):
109
        """
110
        Performs the necessary actions for starting the spawner daemon.
111
        """
112
        # Set the effective user and group to an unprivileged user and group.
113
        self._set_unprivileged_user()
114
115
        # Become a daemon.
116
        # self.__daemonize()
117
118
        # Install signal handlers.
119
        self._install_signal_handlers()
120
121
        self._zmq_init()
122
123
        # Read all user names under which the controller is allowed to start jobs.
124
        JobHandler.read_allowed_users()
125
126
    # ------------------------------------------------------------------------------------------------------------------
127
    def _handle_child_exits(self):
128
        """
129
        Handles an exit of a child and ends a jon handler.
130
        """
131
        try:
132
            pid = -1
133
            while pid != 0:
134
                pid, status = os.waitpid(-1, os.WNOHANG + os.WUNTRACED + os.WCONTINUED)
135
                if pid != 0:
136
                    job_handler = self._job_handlers[pid]
137
138
                    # Send message to controller that a job has finished.
139
                    message = {'type': 'node_stop',
140
                               'sch_id': job_handler.get_sch_id(),
141
                               'rnd_id': job_handler.get_rnd_id(),
142
                               'exit_status': status}
143
                    self._zmq_controller.send_json(message)
144
145
                    # Inform the job handler the job has finished.
146
                    job_handler.set_job_has_finished()
147
148
        except OSError:
0 ignored issues
show
Unused Code introduced by
This except handler seems to be unused and could be removed.

Except handlers which only contain pass and do not have an else clause can usually simply be removed:

try:
    raises_exception()
except:  # Could be removed
    pass
Loading history...
149
            # Ignore OSError. No more children to wait for.
150
            pass
151
152
    # ------------------------------------------------------------------------------------------------------------------
153
    def _event_handler_start_node(self, sch_id, rnd_id, user_name, args):
154
        """
155
        Creates a new job handler and starts the job.
156
157
        :param sch_id: The ID of the schedule of the job.
158
        :param rnd_id: The ID of the job.
159
        :param user_name: The user under which the job must run.
160
        :param args: The arguments for the job.
161
        """
162
        job_handler = JobHandler(sch_id, rnd_id, user_name, args)
163
        job_handler.start_job()
164
165
        self._job_handlers[job_handler.pid] = job_handler
166
167
    # ------------------------------------------------------------------------------------------------------------------
168
    def _read_message(self):
169
        """
170
        Reads a message from the controller.
171
        """
172
        try:
173
            while True:
174
                message = self._zmq_pull_socket.recv_json(zmq.NOBLOCK)
0 ignored issues
show
Bug introduced by
The Module zmq does not seem to have a member named NOBLOCK.

This check looks for calls to members that are non-existent. These calls will fail.

The member could have been renamed or removed.

Loading history...
175
176
                if message['type'] == 'start_node':
177
                    self._event_handler_start_node(message['sch_id'],
178
                                                   message['rnd_id'],
179
                                                   message['user_name'],
180
                                                   message['args'])
181
182
                else:
183
                    raise IndexError("Unknown message type '{0!s}'.".format(message['type']))
184
185
        except zmq.ZMQError as e:
0 ignored issues
show
Coding Style Naming introduced by
The name e does not conform to the variable naming conventions ([a-z_][a-z0-9_]{1,60}$).

This check looks for invalid names for a range of different identifiers.

You can set regular expressions to which the identifiers must conform if the defaults do not match your requirements.

If your project includes a Pylint configuration file, the settings contained in that file take precedence.

To find out more about Pylint, please refer to their site.

Loading history...
Bug introduced by
The Module zmq does not seem to have a member named ZMQError.

This check looks for calls to members that are non-existent. These calls will fail.

The member could have been renamed or removed.

Loading history...
186
            # Ignore ZMQError with EAGAIN. Otherwise, re-raise the error.
187
            if e.errno != zmq.EAGAIN:
0 ignored issues
show
Bug introduced by
The Module zmq does not seem to have a member named EAGAIN.

This check looks for calls to members that are non-existent. These calls will fail.

The member could have been renamed or removed.

Loading history...
188
                raise e
189
190
    # ------------------------------------------------------------------------------------------------------------------
191
    def main(self):
192
        """
193
        The main function of the job spawner.
194
        """
195
        # Perform the necessary actions for starting up the spawner.
196
        self._startup()
197
198
        while True:
199
            # List with all file descriptors for reading.
200
            read = []
201
202
            # Add the queue for incoming messages to the list of read file descriptors.
203
            zmq_fd = self._zmq_pull_socket.get(zmq.FD)
0 ignored issues
show
Bug introduced by
The Module zmq does not seem to have a member named FD.

This check looks for calls to members that are non-existent. These calls will fail.

The member could have been renamed or removed.

Loading history...
204
            read.append(zmq_fd)
205
206
            # Add the job handlers to the list of read file descriptors.
207
            remove = []
208
            for pid, job_handler in self._job_handlers.items():
209
                fd_stdout = job_handler.get_stdout()
210
                if fd_stdout >= 0:
211
                    read.append(fd_stdout)
212
                fd_stderr = job_handler.get_stderr()
213
                if fd_stderr >= 0:
214
                    read.append(fd_stderr)
215
216
                if fd_stdout == -1 and fd_stderr == -1 and job_handler.get_pid() == -1:
217
                    # The job handler has read all data from stdout and stderr from job and the child process has
218
                    # exited.
219
                    remove.append(pid)
220
221
            # Remove jobs that are finished.
222
            for pid in remove:
223
                job_handler = self._job_handlers[pid]
224
225
                # Tell the job handler we are done with the job.
226
                job_handler.end_job()
227
228
                # Send messages to logger daemon that the stdout and stderr of the job can be loaded into
229
                # the database.
230
                self._zmq_logger.send_json(job_handler.get_logger_message('out'))
231
                self._zmq_logger.send_json(job_handler.get_logger_message('err'))
232
233
                # Remove the job from the dictionary with jobs.
234
                del self._job_handlers[pid]
235
236
            # Unblock interrupts.
237
            signal.pthread_sigmask(signal.SIG_UNBLOCK, {signal.SIGCHLD, signal.SIGHUP})
238
239
            try:
240
                # Wait for a fd becomes available for read or wait for an interrupt.
241
                read, _, _ = select.select(read, [], [])
242
243
            except InterruptedError:
0 ignored issues
show
Unused Code introduced by
This except handler seems to be unused and could be removed.

Except handlers which only contain pass and do not have an else clause can usually simply be removed:

try:
    raises_exception()
except:  # Could be removed
    pass
Loading history...
244
                # Ignore Interrupted system call errors (EINTR) in the select call.
245
                pass
246
247
            # Block all interrupts to prevent interrupted system call errors (EINTR).
248
            signal.pthread_sigmask(signal.SIG_BLOCK, {signal.SIGCHLD, signal.SIGHUP})
249
250
            for fd in read:
251
                if fd == zmq_fd:
252
                    # fd of the message queue is ready to receive data.
253
                    self._read_message()
254
                else:
255
                    # fd of one or more job handlers are ready to receive data.
256
                    for _, job_handler in self._job_handlers.items():
257
                        if fd == job_handler.get_stdout():
258
                            job_handler.read(fd)
259
                        if fd == job_handler.get_stderr():
260
                            job_handler.read(fd)
261
262
            if self._child_flag:
263
                # Process one or more exited child processes.
264
                self._handle_child_exits()
265
266
267
# ----------------------------------------------------------------------------------------------------------------------
268