Completed
Pull Request — master (#73)
by Helpful
02:27
created

QueuedJobService::grabMutex()   B

Complexity

Conditions 4
Paths 3

Size

Total Lines 22
Code Lines 11

Duplication

Lines 0
Ratio 0 %
Metric Value
dl 0
loc 22
rs 8.9197
cc 4
eloc 11
nc 3
nop 1
1
<?php
2
3
/**
4
 * A service that can be used for starting, stopping and listing queued jobs.
5
 *
6
 * When a job is first added, it is initialised, its job type determined, then persisted to the database
7
 *
8
 * When the queues are scanned, a job is reloaded and processed. Ignoring the persistence and reloading, it looks
9
 * something like
10
 *
11
 * job->getJobType();
12
 * job->getJobData();
13
 * data->write();
14
 * job->setup();
15
 * while !job->isComplete
16
 *	job->process();
17
 *	job->getJobData();
18
 *  data->write();
19
 *
20
 *
21
 * @author Marcus Nyeholt <[email protected]>
22
 * @license BSD http://silverstripe.org/bsd-license/
23
 */
24
class QueuedJobService
0 ignored issues
show
Coding Style Compatibility introduced by
PSR1 recommends that each class must be in a namespace of at least one level to avoid collisions.

You can fix this by adding a namespace to your class:

namespace YourVendor;

class YourClass { }

When choosing a vendor namespace, try to pick something that is not too generic to avoid conflicts with other libraries.

Loading history...
25
{
26
    /**
27
     * @var int
28
     */
29
    private static $stall_threshold = 3;
0 ignored issues
show
Unused Code introduced by
The property $stall_threshold is not used and could be removed.

This check marks private properties in classes that are never used. Those properties can be removed.

Loading history...
30
31
    /**
32
     * How much ram will we allow before pausing and releasing the memory?
33
     *
34
     * For instance, set to 134217728 (128MB) to pause this process if used memory exceeds
35
     * this value. This needs to be set to a value lower than the php_ini max_memory as
36
     * the system will otherwise crash before shutdown can be handled gracefully.
37
     *
38
     * @var int
39
     * @config
40
     */
41
    private static $memory_limit = 134217728;
0 ignored issues
show
Unused Code introduced by
The property $memory_limit is not used and could be removed.

This check marks private properties in classes that are never used. Those properties can be removed.

Loading history...
42
43
    /**
44
     * Optional time limit (in seconds) to run the service before restarting to release resources.
45
     *
46
     * Defaults to no limit.
47
     *
48
     * @var int
49
     * @config
50
     */
51
    private static $time_limit = 0;
0 ignored issues
show
Unused Code introduced by
The property $time_limit is not used and could be removed.

This check marks private properties in classes that are never used. Those properties can be removed.

Loading history...
52
53
    /**
54
     * Timestamp (in seconds) when the queue was started
55
     *
56
     * @var int
57
     */
58
    protected $startedAt = 0;
59
60
    /**
61
     * Should "immediate" jobs be managed using the shutdown function?
62
     *
63
     * It is recommended you set up an inotify watch and use that for
64
     * triggering immediate jobs. See the wiki for more information
65
     *
66
     * @var boolean
67
     */
68
    private static $use_shutdown_function = true;
0 ignored issues
show
Unused Code introduced by
The property $use_shutdown_function is not used and could be removed.

This check marks private properties in classes that are never used. Those properties can be removed.

Loading history...
69
70
    /**
71
     * The location for immediate jobs to be stored in
72
     *
73
     * @var string
74
     */
75
    private static $cache_dir = 'queuedjobs';
0 ignored issues
show
Unused Code introduced by
The property $cache_dir is not used and could be removed.

This check marks private properties in classes that are never used. Those properties can be removed.

Loading history...
76
77
    /**
78
     * @var DefaultQueueHandler
79
     */
80
    public $queueHandler;
81
82
    /**
83
     *
84
     * @var TaskRunnerEngine
85
     */
86
    public $queuedRunner;
87
88
    /**
89
     * Register our shutdown handler
90
     */
91
    public function __construct()
92
    {
93
        // bind a shutdown function to process all 'immediate' queued jobs if needed, but only in CLI mode
94
        if (Config::inst()->get(__CLASS__, 'use_shutdown_function') && Director::is_cli()) {
95
            if (class_exists('PHPUnit_Framework_TestCase') && SapphireTest::is_running_test()) {
0 ignored issues
show
Unused Code introduced by
This if statement is empty and can be removed.

This check looks for the bodies of if statements that have no statements or where all statements have been commented out. This may be the result of changes for debugging or the code may simply be obsolete.

These if bodies can be removed. If you have an empty if but statements in the else branch, consider inverting the condition.

if (rand(1, 6) > 3) {
//print "Check failed";
} else {
    print "Check succeeded";
}

could be turned into

if (rand(1, 6) <= 3) {
    print "Check succeeded";
}

This is much more concise to read.

Loading history...
96
                // do NOTHING
97
            } else {
98
                register_shutdown_function(array($this, 'onShutdown'));
99
            }
100
        }
101
        if (Config::inst()->get('Email', 'queued_jobs_admin_email') == '') {
102
            Config::inst()->update('Email', 'queued_jobs_admin_email', Config::inst()->get('Email', 'admin_email'));
103
        }
104
    }
105
106
    /**
107
     * Adds a job to the queue to be started
108
     *
109
     * Relevant data about the job will be persisted using a QueuedJobDescriptor
110
     *
111
     * @param QueuedJob $job
112
     *			The job to start.
113
     * @param $startAfter
114
     *			The date (in Y-m-d H:i:s format) to start execution after
115
     * @param int $userId
116
     *			The ID of a user to execute the job as. Defaults to the current user
117
     * @return int
118
     */
119
    public function queueJob(QueuedJob $job, $startAfter = null, $userId = null, $queueName = null)
120
    {
121
        $signature = $job->getSignature();
122
123
        // see if we already have this job in a queue
124
        $filter = array(
125
            'Signature' => $signature,
126
            'JobStatus' => array(
127
                QueuedJob::STATUS_NEW,
128
                QueuedJob::STATUS_INIT
129
            )
130
        );
131
132
        $existing = DataList::create('QueuedJobDescriptor')->filter($filter)->first();
133
134
        if ($existing && $existing->ID) {
135
            return $existing->ID;
136
        }
137
138
        $jobDescriptor = new QueuedJobDescriptor();
139
        $jobDescriptor->JobTitle = $job->getTitle();
140
        $jobDescriptor->JobType = $queueName ? $queueName : $job->getJobType();
141
        $jobDescriptor->Signature = $signature;
142
        $jobDescriptor->Implementation = get_class($job);
143
        $jobDescriptor->StartAfter = $startAfter;
144
145
        $jobDescriptor->RunAsID = $userId ? $userId : Member::currentUserID();
0 ignored issues
show
Documentation introduced by
The property RunAsID does not exist on object<QueuedJobDescriptor>. Since you implemented __set, maybe consider adding a @property annotation.

Since your code implements the magic setter _set, this function will be called for any write access on an undefined variable. You can add the @property annotation to your class or interface to document the existence of this variable.

<?php

/**
 * @property int $x
 * @property int $y
 * @property string $text
 */
class MyLabel
{
    private $properties;

    private $allowedProperties = array('x', 'y', 'text');

    public function __get($name)
    {
        if (isset($properties[$name]) && in_array($name, $this->allowedProperties)) {
            return $properties[$name];
        } else {
            return null;
        }
    }

    public function __set($name, $value)
    {
        if (in_array($name, $this->allowedProperties)) {
            $properties[$name] = $value;
        } else {
            throw new \LogicException("Property $name is not defined.");
        }
    }

}

Since the property has write access only, you can use the @property-write annotation instead.

Of course, you may also just have mistyped another name, in which case you should fix the error.

See also the PhpDoc documentation for @property.

Loading history...
146
147
        // copy data
148
        $this->copyJobToDescriptor($job, $jobDescriptor);
0 ignored issues
show
Documentation introduced by
$jobDescriptor is of type object<QueuedJobDescriptor>, but the function expects a object<JobDescriptor>.

It seems like the type of the argument is not accepted by the function/method which you are calling.

In some cases, in particular if PHP’s automatic type-juggling kicks in this might be fine. In other cases, however this might be a bug.

We suggest to add an explicit type cast like in the following example:

function acceptsInteger($int) { }

$x = '123'; // string "123"

// Instead of
acceptsInteger($x);

// we recommend to use
acceptsInteger((integer) $x);
Loading history...
149
150
        $jobDescriptor->write();
151
152
        $this->startJob($jobDescriptor, $startAfter);
0 ignored issues
show
Documentation introduced by
$jobDescriptor is of type object<QueuedJobDescriptor>, but the function expects a object<JobDescriptor>.

It seems like the type of the argument is not accepted by the function/method which you are calling.

In some cases, in particular if PHP’s automatic type-juggling kicks in this might be fine. In other cases, however this might be a bug.

We suggest to add an explicit type cast like in the following example:

function acceptsInteger($int) { }

$x = '123'; // string "123"

// Instead of
acceptsInteger($x);

// we recommend to use
acceptsInteger((integer) $x);
Loading history...
153
154
        return $jobDescriptor->ID;
155
    }
156
157
    /**
158
     * Start a job (or however the queue handler determines it should be started)
159
     *
160
     * @param JobDescriptor $jobDescriptor
161
     * @param date $startAfter
162
     */
163
    public function startJob($jobDescriptor, $startAfter = null)
164
    {
165
        if ($startAfter && strtotime($startAfter) > time()) {
166
            $this->queueHandler->scheduleJob($jobDescriptor, $startAfter);
167
        } else {
168
            // immediately start it on the queue, however that works
169
            $this->queueHandler->startJobOnQueue($jobDescriptor);
170
        }
171
    }
172
173
    /**
174
     * Copies data from a job into a descriptor for persisting
175
     *
176
     * @param QueuedJob $job
177
     * @param JobDescriptor $jobDescriptor
178
     */
179
    protected function copyJobToDescriptor($job, $jobDescriptor)
180
    {
181
        $data = $job->getJobData();
182
183
        $jobDescriptor->TotalSteps = $data->totalSteps;
184
        $jobDescriptor->StepsProcessed = $data->currentStep;
185
        if ($data->isComplete) {
186
            $jobDescriptor->JobStatus = QueuedJob::STATUS_COMPLETE;
187
            $jobDescriptor->JobFinished = date('Y-m-d H:i:s');
188
        }
189
190
        $jobDescriptor->SavedJobData = serialize($data->jobData);
191
        $jobDescriptor->SavedJobMessages = serialize($data->messages);
192
    }
193
194
    /**
195
     * @param QueuedJobDescriptor $jobDescriptor
196
     * @param QueuedJob $job
197
     */
198
    protected function copyDescriptorToJob($jobDescriptor, $job)
199
    {
200
        $jobData = null;
0 ignored issues
show
Unused Code introduced by
$jobData is not used, you could remove the assignment.

This check looks for variable assignements that are either overwritten by other assignments or where the variable is not used subsequently.

$myVar = 'Value';
$higher = false;

if (rand(1, 6) > 3) {
    $higher = true;
} else {
    $higher = false;
}

Both the $myVar assignment in line 1 and the $higher assignment in line 2 are dead. The first because $myVar is never used and the second because $higher is always overwritten for every possible time line.

Loading history...
201
        $messages = null;
0 ignored issues
show
Unused Code introduced by
$messages is not used, you could remove the assignment.

This check looks for variable assignements that are either overwritten by other assignments or where the variable is not used subsequently.

$myVar = 'Value';
$higher = false;

if (rand(1, 6) > 3) {
    $higher = true;
} else {
    $higher = false;
}

Both the $myVar assignment in line 1 and the $higher assignment in line 2 are dead. The first because $myVar is never used and the second because $higher is always overwritten for every possible time line.

Loading history...
202
203
        // switching to php's serialize methods... not sure why this wasn't done from the start!
204
        $jobData = @unserialize($jobDescriptor->SavedJobData);
205
        $messages = @unserialize($jobDescriptor->SavedJobMessages);
206
207
        if (!$jobData) {
208
            // SS's convert:: function doesn't do this detection for us!!
209
            if (function_exists('json_decode')) {
210
                $jobData = json_decode($jobDescriptor->SavedJobData);
211
                $messages = json_decode($jobDescriptor->SavedJobMessages);
212
            } else {
213
                $jobData = Convert::json2obj($jobDescriptor->SavedJobData);
214
                $messages = Convert::json2obj($jobDescriptor->SavedJobMessages);
215
            }
216
        }
217
218
        $job->setJobData(
219
            $jobDescriptor->TotalSteps,
220
            $jobDescriptor->StepsProcessed,
221
            $jobDescriptor->JobStatus == QueuedJob::STATUS_COMPLETE,
222
            $jobData,
223
            $messages
224
        );
225
    }
226
227
    /**
228
     * Check the current job queues and see if any of the jobs currently in there should be started. If so,
229
     * return the next job that should be executed
230
     *
231
     * @param string $type Job type
232
     * @return QueuedJobDescriptor
233
     */
234
    public function getNextPendingJob($type = null)
235
    {
236
        // Filter jobs by type
237
        $type = $type ?: QueuedJob::QUEUED;
238
        $list = QueuedJobDescriptor::get()
239
            ->filter('JobType', $type)
240
            ->sort('ID', 'ASC');
241
242
        // see if there's any blocked jobs that need to be resumed
243
        $waitingJob = $list
244
            ->filter('JobStatus', QueuedJob::STATUS_WAIT)
245
            ->first();
246
        if ($waitingJob) {
247
            return $waitingJob;
248
        }
249
250
        // If there's an existing job either running or pending, the lets just return false to indicate
251
        // that we're still executing
252
        $runningJob = $list
253
            ->filter('JobStatus', array(QueuedJob::STATUS_INIT, QueuedJob::STATUS_RUN))
254
            ->first();
255
        if ($runningJob) {
256
            return false;
0 ignored issues
show
Bug Best Practice introduced by
The return type of return false; (false) is incompatible with the return type documented by QueuedJobService::getNextPendingJob of type QueuedJobDescriptor.

If you return a value from a function or method, it should be a sub-type of the type that is given by the parent type f.e. an interface, or abstract method. This is more formally defined by the Lizkov substitution principle, and guarantees that classes that depend on the parent type can use any instance of a child type interchangably. This principle also belongs to the SOLID principles for object oriented design.

Let’s take a look at an example:

class Author {
    private $name;

    public function __construct($name) {
        $this->name = $name;
    }

    public function getName() {
        return $this->name;
    }
}

abstract class Post {
    public function getAuthor() {
        return 'Johannes';
    }
}

class BlogPost extends Post {
    public function getAuthor() {
        return new Author('Johannes');
    }
}

class ForumPost extends Post { /* ... */ }

function my_function(Post $post) {
    echo strtoupper($post->getAuthor());
}

Our function my_function expects a Post object, and outputs the author of the post. The base class Post returns a simple string and outputting a simple string will work just fine. However, the child class BlogPost which is a sub-type of Post instead decided to return an object, and is therefore violating the SOLID principles. If a BlogPost were passed to my_function, PHP would not complain, but ultimately fail when executing the strtoupper call in its body.

Loading history...
257
        }
258
259
        // Otherwise, lets find any 'new' jobs that are waiting to execute
260
        $newJob = $list
261
            ->filter('JobStatus', QueuedJob::STATUS_NEW)
262
            ->where(sprintf(
263
                '"StartAfter" < \'%s\' OR "StartAfter" IS NULL',
264
                SS_DateTime::now()->getValue()
265
            ))
266
            ->first();
267
268
        return $newJob;
269
    }
270
271
    /**
272
     * Runs an explicit check on all currently running jobs to make sure their "processed" count is incrementing
273
     * between each run. If it's not, then we need to flag it as paused due to an error.
274
     *
275
     * This typically happens when a PHP fatal error is thrown, which can't be picked up by the error
276
     * handler or exception checker; in this case, we detect these stalled jobs later and fix (try) to
277
     * fix them
278
     */
279
    public function checkJobHealth()
280
    {
281
        // Select all jobs currently marked as running
282
        $runningJobs = QueuedJobDescriptor::get()
283
            ->filter(
284
                'JobStatus',
285
                array(
286
                    QueuedJob::STATUS_RUN,
287
                    QueuedJob::STATUS_INIT
288
                )
289
            );
290
291
        // If no steps have been processed since the last run, consider it a broken job
292
        // Only check jobs that have been viewed before. LastProcessedCount defaults to -1 on new jobs.
293
        $stalledJobs = $runningJobs
294
            ->filter('LastProcessedCount:GreaterThanOrEqual', 0)
295
            ->where('"StepsProcessed" = "LastProcessedCount"');
296
        foreach ($stalledJobs as $stalledJob) {
297
            $this->restartStalledJob($stalledJob);
298
        }
299
300
        // now, find those that need to be marked before the next check
301
        // foreach job, mark it as having been incremented
302
        foreach ($runningJobs as $job) {
303
            $job->LastProcessedCount = $job->StepsProcessed;
304
            $job->write();
305
        }
306
307
        // finally, find the list of broken jobs and send an email if there's some found
308
        $brokenJobs = QueuedJobDescriptor::get()->filter('JobStatus', QueuedJob::STATUS_BROKEN);
309 View Code Duplication
        if ($brokenJobs && $brokenJobs->count()) {
0 ignored issues
show
Duplication introduced by
This code seems to be duplicated across your project.

Duplicated code is one of the most pungent code smells. If you need to duplicate the same code in three or more different places, we strongly encourage you to look into extracting the code into a single class or operation.

You can also find more detailed suggestions in the “Code” section of your repository.

Loading history...
310
            SS_Log::log(array(
311
                'errno' => 0,
312
                'errstr' => 'Broken jobs were found in the job queue',
313
                'errfile' => __FILE__,
314
                'errline' => __LINE__,
315
                'errcontext' => ''
316
            ), SS_Log::ERR);
317
        }
318
    }
319
320
    /**
321
     * Attempt to restart a stalled job
322
     *
323
     * @param QueuedJobDescriptor $stalledJob
324
     * @return bool True if the job was successfully restarted
325
     */
326
    protected function restartStalledJob($stalledJob)
327
    {
328
        if ($stalledJob->ResumeCounts < Config::inst()->get(__CLASS__, 'stall_threshold')) {
329
            $stalledJob->restart();
330
            $message = sprintf(
331
                _t(
332
                    'QueuedJobs.STALLED_JOB_MSG',
333
                    'A job named %s appears to have stalled. It will be stopped and restarted, please login to make sure it has continued'
334
                ),
335
                $stalledJob->JobTitle
336
            );
337
        } else {
338
            $stalledJob->pause();
339
            $message = sprintf(
340
                _t(
341
                    'QueuedJobs.STALLED_JOB_MSG',
342
                    'A job named %s appears to have stalled. It has been paused, please login to check it'
343
                ),
344
                $stalledJob->JobTitle
345
            );
346
        }
347
348
        singleton('QJUtils')->log($message);
349
        $from = Config::inst()->get('Email', 'admin_email');
350
        $to = Config::inst()->get('Email', 'queued_job_admin_email');
351
        $subject = _t('QueuedJobs.STALLED_JOB', 'Stalled job');
352
        $mail = new Email($from, $to, $subject, $message);
353
        $mail->send();
354
    }
355
356
    /**
357
     * Prepares the given jobDescriptor for execution. Returns the job that
358
     * will actually be run in a state ready for executing.
359
     *
360
     * Note that this is called each time a job is picked up to be executed from the cron
361
     * job - meaning that jobs that are paused and restarted will have 'setup()' called on them again,
362
     * so your job MUST detect that and act accordingly.
363
     *
364
     * @param QueuedJobDescriptor $jobDescriptor
365
     *			The Job descriptor of a job to prepare for execution
366
     *
367
     * @return QueuedJob|boolean
368
     */
369
    protected function initialiseJob(QueuedJobDescriptor $jobDescriptor)
370
    {
371
        // create the job class
372
        $impl = $jobDescriptor->Implementation;
373
        $job = Object::create($impl);
374
        /* @var $job QueuedJob */
375
        if (!$job) {
376
            throw new Exception("Implementation $impl no longer exists");
377
        }
378
379
        $jobDescriptor->JobStatus = QueuedJob::STATUS_INIT;
380
        $jobDescriptor->write();
381
382
        // make sure the data is there
383
        $this->copyDescriptorToJob($jobDescriptor, $job);
384
385
        // see if it needs 'setup' or 'restart' called
386
        if ($jobDescriptor->StepsProcessed <= 0) {
387
            $job->setup();
388
        } else {
389
            $job->prepareForRestart();
390
        }
391
392
        // make sure the descriptor is up to date with anything changed
393
        $this->copyJobToDescriptor($job, $jobDescriptor);
0 ignored issues
show
Documentation introduced by
$jobDescriptor is of type object<QueuedJobDescriptor>, but the function expects a object<JobDescriptor>.

It seems like the type of the argument is not accepted by the function/method which you are calling.

In some cases, in particular if PHP’s automatic type-juggling kicks in this might be fine. In other cases, however this might be a bug.

We suggest to add an explicit type cast like in the following example:

function acceptsInteger($int) { }

$x = '123'; // string "123"

// Instead of
acceptsInteger($x);

// we recommend to use
acceptsInteger((integer) $x);
Loading history...
394
        $jobDescriptor->write();
395
396
        return $job;
397
    }
398
399
    /**
400
     * Given a {@link QueuedJobDescriptor} mark the job as initialised. Works sort of like a mutex.
401
     * Currently a database lock isn't entirely achievable, due to database adapters not supporting locks.
402
     * This may still have a race condition, but this should minimise the possibility.
403
     * Side effect is the job status will be changed to "Initialised".
404
     *
405
     * Assumption is the job has a status of "Queued" or "Wait".
406
     *
407
     * @param QueuedJobDescriptor $jobDescriptor
408
     * @return boolean
409
     */
410
    protected function grabMutex(QueuedJobDescriptor $jobDescriptor)
411
    {
412
        // write the status and determine if any rows were affected, for protection against a
413
        // potential race condition where two or more processes init the same job at once.
414
        // This deliberately does not use write() as that would always update LastEdited
415
        // and thus the row would always be affected.
416
        try {
417
            DB::query(sprintf(
418
                'UPDATE "QueuedJobDescriptor" SET "JobStatus" = \'%s\' WHERE "ID" = %s',
419
                QueuedJob::STATUS_INIT,
420
                $jobDescriptor->ID
421
            ));
422
        } catch (Exception $e) {
423
            return false;
424
        }
425
426
        if (DB::getConn()->affectedRows() === 0 && $jobDescriptor->JobStatus !== QueuedJob::STATUS_INIT) {
0 ignored issues
show
Deprecated Code introduced by
The method DB::getConn() has been deprecated with message: since version 4.0 Use DB::get_conn instead

This method has been deprecated. The supplier of the class has supplied an explanatory message.

The explanatory message should give you some clue as to whether and when the method will be removed from the class and what other method or class to use instead.

Loading history...
427
            return false;
428
        }
429
430
        return true;
431
    }
432
433
    /**
434
     * Start the actual execution of a job.
435
     * The assumption is the jobID refers to a {@link QueuedJobDescriptor} that is status set as "Queued".
436
     *
437
     * This method will continue executing until the job says it's completed
438
     *
439
     * @param int $jobId
440
     *			The ID of the job to start executing
441
     * @return boolean
442
     */
443
    public function runJob($jobId)
0 ignored issues
show
Coding Style introduced by
runJob uses the super-global variable $_SESSION which is generally not recommended.

Instead of super-globals, we recommend to explicitly inject the dependencies of your class. This makes your code less dependent on global state and it becomes generally more testable:

// Bad
class Router
{
    public function generate($path)
    {
        return $_SERVER['HOST'].$path;
    }
}

// Better
class Router
{
    private $host;

    public function __construct($host)
    {
        $this->host = $host;
    }

    public function generate($path)
    {
        return $this->host.$path;
    }
}

class Controller
{
    public function myAction(Request $request)
    {
        // Instead of
        $page = isset($_GET['page']) ? intval($_GET['page']) : 1;

        // Better (assuming you use the Symfony2 request)
        $page = $request->query->get('page', 1);
    }
}
Loading history...
444
    {
445
        // first retrieve the descriptor
446
        $jobDescriptor = DataObject::get_by_id('QueuedJobDescriptor', (int) $jobId);
447
        if (!$jobDescriptor) {
448
            throw new Exception("$jobId is invalid");
449
        }
450
451
        // now lets see whether we have a current user to run as. Typically, if the job is executing via the CLI,
452
        // we want it to actually execute as the RunAs user - however, if running via the web (which is rare...), we
453
        // want to ensure that the current user has admin privileges before switching. Otherwise, we just run it
454
        // as the currently logged in user and hope for the best
455
456
        // We need to use $_SESSION directly because SS ties the session to a controller that no longer exists at
457
        // this point of execution in some circumstances
458
        $originalUserID = isset($_SESSION['loggedInAs']) ? $_SESSION['loggedInAs'] : 0;
459
        $originalUser = $originalUserID ? DataObject::get_by_id('Member', $originalUserID) : null;
460
        $runAsUser = null;
461
462
        if (Director::is_cli() || !$originalUser || Permission::checkMember($originalUser, 'ADMIN')) {
463
            $runAsUser = $jobDescriptor->RunAs();
464
            if ($runAsUser && $runAsUser->exists()) {
465
                // the job runner outputs content way early in the piece, meaning there'll be cookie errors
466
                // if we try and do a normal login, and we only want it temporarily...
467
                if (Controller::has_curr()) {
468
                    Session::set('loggedInAs', $runAsUser->ID);
469
                } else {
470
                    $_SESSION['loggedInAs'] = $runAsUser->ID;
471
                }
472
473
                // this is an explicit coupling brought about by SS not having
474
                // a nice way of mocking a user, as it requires session
475
                // nastiness
476
                if (class_exists('SecurityContext')) {
477
                    singleton('SecurityContext')->setMember($runAsUser);
478
                }
479
            }
480
        }
481
482
        // set up a custom error handler for this processing
483
        $errorHandler = new JobErrorHandler();
484
485
        $job = null;
486
487
        $broken = false;
488
489
        // Push a config context onto the stack for the duration of this job run.
490
        Config::nest();
491
492
        if ($this->grabMutex($jobDescriptor)) {
0 ignored issues
show
Compatibility introduced by
$jobDescriptor of type object<DataObject> is not a sub-type of object<QueuedJobDescriptor>. It seems like you assume a child class of the class DataObject to be always present.

This check looks for parameters that are defined as one type in their type hint or doc comment but seem to be used as a narrower type, i.e an implementation of an interface or a subclass.

Consider changing the type of the parameter or doing an instanceof check before assuming your parameter is of the expected type.

Loading history...
493
            try {
494
                $job = $this->initialiseJob($jobDescriptor);
0 ignored issues
show
Compatibility introduced by
$jobDescriptor of type object<DataObject> is not a sub-type of object<QueuedJobDescriptor>. It seems like you assume a child class of the class DataObject to be always present.

This check looks for parameters that are defined as one type in their type hint or doc comment but seem to be used as a narrower type, i.e an implementation of an interface or a subclass.

Consider changing the type of the parameter or doing an instanceof check before assuming your parameter is of the expected type.

Loading history...
495
496
                // get the job ready to begin.
497
                if (!$jobDescriptor->JobStarted) {
498
                    $jobDescriptor->JobStarted = date('Y-m-d H:i:s');
499
                } else {
500
                    $jobDescriptor->JobRestarted = date('Y-m-d H:i:s');
501
                }
502
503
                $jobDescriptor->JobStatus = QueuedJob::STATUS_RUN;
504
                $jobDescriptor->write();
505
506
                $lastStepProcessed = 0;
507
                // have we stalled at all?
508
                $stallCount = 0;
509
510
                if ($job->SubsiteID && class_exists('Subsite')) {
0 ignored issues
show
Bug introduced by
Accessing SubsiteID on the interface QueuedJob suggest that you code against a concrete implementation. How about adding an instanceof check?

If you access a property on an interface, you most likely code against a concrete implementation of the interface.

Available Fixes

  1. Adding an additional type check:

    interface SomeInterface { }
    class SomeClass implements SomeInterface {
        public $a;
    }
    
    function someFunction(SomeInterface $object) {
        if ($object instanceof SomeClass) {
            $a = $object->a;
        }
    }
    
  2. Changing the type hint:

    interface SomeInterface { }
    class SomeClass implements SomeInterface {
        public $a;
    }
    
    function someFunction(SomeClass $object) {
        $a = $object->a;
    }
    
Loading history...
511
                    Subsite::changeSubsite($job->SubsiteID);
0 ignored issues
show
Bug introduced by
Accessing SubsiteID on the interface QueuedJob suggest that you code against a concrete implementation. How about adding an instanceof check?

If you access a property on an interface, you most likely code against a concrete implementation of the interface.

Available Fixes

  1. Adding an additional type check:

    interface SomeInterface { }
    class SomeClass implements SomeInterface {
        public $a;
    }
    
    function someFunction(SomeInterface $object) {
        if ($object instanceof SomeClass) {
            $a = $object->a;
        }
    }
    
  2. Changing the type hint:

    interface SomeInterface { }
    class SomeClass implements SomeInterface {
        public $a;
    }
    
    function someFunction(SomeClass $object) {
        $a = $object->a;
    }
    
Loading history...
512
513
                    // lets set the base URL as far as Director is concerned so that our URLs are correct
514
                    $subsite = DataObject::get_by_id('Subsite', $job->SubsiteID);
0 ignored issues
show
Bug introduced by
Accessing SubsiteID on the interface QueuedJob suggest that you code against a concrete implementation. How about adding an instanceof check?

If you access a property on an interface, you most likely code against a concrete implementation of the interface.

Available Fixes

  1. Adding an additional type check:

    interface SomeInterface { }
    class SomeClass implements SomeInterface {
        public $a;
    }
    
    function someFunction(SomeInterface $object) {
        if ($object instanceof SomeClass) {
            $a = $object->a;
        }
    }
    
  2. Changing the type hint:

    interface SomeInterface { }
    class SomeClass implements SomeInterface {
        public $a;
    }
    
    function someFunction(SomeClass $object) {
        $a = $object->a;
    }
    
Loading history...
515
                    if ($subsite && $subsite->exists()) {
516
                        $domain = $subsite->domain();
517
                        $base = rtrim(Director::protocol() . $domain, '/') . '/';
518
519
                        Config::inst()->update('Director', 'alternate_base_url', $base);
520
                    }
521
                }
522
523
                // while not finished
524
                while (!$job->jobFinished() && !$broken) {
525
                    // see that we haven't been set to 'paused' or otherwise by another process
526
                    $jobDescriptor = DataObject::get_by_id('QueuedJobDescriptor', (int) $jobId);
527
                    if (!$jobDescriptor || !$jobDescriptor->exists()) {
528
                        $broken = true;
529
                        SS_Log::log(array(
530
                            'errno' => 0,
531
                            'errstr' => 'Job descriptor ' . $jobId . ' could not be found',
532
                            'errfile' => __FILE__,
533
                            'errline' => __LINE__,
534
                            'errcontext' => ''
535
                        ), SS_Log::ERR);
536
                        break;
537
                    }
538
                    if ($jobDescriptor->JobStatus != QueuedJob::STATUS_RUN) {
539
                        // we've been paused by something, so we'll just exit
540
                        $job->addMessage(sprintf(_t('QueuedJobs.JOB_PAUSED', "Job paused at %s"), date('Y-m-d H:i:s')));
541
                        $broken = true;
542
                    }
543
544
                    if (!$broken) {
545
                        try {
546
                            $job->process();
547
                        } catch (Exception $e) {
548
                            // okay, we'll just catch this exception for now
549
                            $job->addMessage(sprintf(_t('QueuedJobs.JOB_EXCEPT', 'Job caused exception %s in %s at line %s'), $e->getMessage(), $e->getFile(), $e->getLine()), 'ERROR');
0 ignored issues
show
Unused Code introduced by
The call to QueuedJob::addMessage() has too many arguments starting with 'ERROR'.

This check compares calls to functions or methods with their respective definitions. If the call has more arguments than are defined, it raises an issue.

If a function is defined several times with a different number of parameters, the check may pick up the wrong definition and report false positives. One codebase where this has been known to happen is Wordpress.

In this case you can add the @ignore PhpDoc annotation to the duplicate definition and it will be ignored.

Loading history...
550
                            SS_Log::log($e, SS_Log::ERR);
551
                            $jobDescriptor->JobStatus =  QueuedJob::STATUS_BROKEN;
552
                        }
553
554
                        // now check the job state
555
                        $data = $job->getJobData();
556
                        if ($data->currentStep == $lastStepProcessed) {
557
                            $stallCount++;
558
                        }
559
560
                        if ($stallCount > Config::inst()->get(__CLASS__, 'stall_threshold')) {
561
                            $broken = true;
562
                            $job->addMessage(sprintf(_t('QueuedJobs.JOB_STALLED', "Job stalled after %s attempts - please check"), $stallCount), 'ERROR');
0 ignored issues
show
Unused Code introduced by
The call to QueuedJob::addMessage() has too many arguments starting with 'ERROR'.

This check compares calls to functions or methods with their respective definitions. If the call has more arguments than are defined, it raises an issue.

If a function is defined several times with a different number of parameters, the check may pick up the wrong definition and report false positives. One codebase where this has been known to happen is Wordpress.

In this case you can add the @ignore PhpDoc annotation to the duplicate definition and it will be ignored.

Loading history...
563
                            $jobDescriptor->JobStatus =  QueuedJob::STATUS_BROKEN;
564
                        }
565
566
                        // now we'll be good and check our memory usage. If it is too high, we'll set the job to
567
                        // a 'Waiting' state, and let the next processing run pick up the job.
568
                        if ($this->isMemoryTooHigh()) {
569
                            $job->addMessage(sprintf(
570
                                _t('QueuedJobs.MEMORY_RELEASE', 'Job releasing memory and waiting (%s used)'),
571
                                $this->humanReadable($this->getMemoryUsage())
572
                            ));
573
                            $jobDescriptor->JobStatus = QueuedJob::STATUS_WAIT;
574
                            $broken = true;
575
                        }
576
577
                        // Also check if we are running too long
578
                        if ($this->hasPassedTimeLimit()) {
579
                            $job->addMessage(_t(
580
                                'QueuedJobs.TIME_LIMIT',
581
                                'Queue has passed time limit and will restart before continuing'
582
                            ));
583
                            $jobDescriptor->JobStatus = QueuedJob::STATUS_WAIT;
584
                            $broken = true;
585
                        }
586
                    }
587
588
                    if ($jobDescriptor) {
589
                        $this->copyJobToDescriptor($job, $jobDescriptor);
0 ignored issues
show
Documentation introduced by
$jobDescriptor is of type object<DataObject>, but the function expects a object<JobDescriptor>.

It seems like the type of the argument is not accepted by the function/method which you are calling.

In some cases, in particular if PHP’s automatic type-juggling kicks in this might be fine. In other cases, however this might be a bug.

We suggest to add an explicit type cast like in the following example:

function acceptsInteger($int) { }

$x = '123'; // string "123"

// Instead of
acceptsInteger($x);

// we recommend to use
acceptsInteger((integer) $x);
Loading history...
590
                        $jobDescriptor->write();
591 View Code Duplication
                    } else {
0 ignored issues
show
Duplication introduced by
This code seems to be duplicated across your project.

Duplicated code is one of the most pungent code smells. If you need to duplicate the same code in three or more different places, we strongly encourage you to look into extracting the code into a single class or operation.

You can also find more detailed suggestions in the “Code” section of your repository.

Loading history...
592
                        SS_Log::log(array(
593
                            'errno' => 0,
594
                            'errstr' => 'Job descriptor has been set to null',
595
                            'errfile' => __FILE__,
596
                            'errline' => __LINE__,
597
                            'errcontext' => ''
598
                        ), SS_Log::WARN);
599
                        $broken = true;
600
                    }
601
                }
602
603
                // a last final save. The job is complete by now
604
                if ($jobDescriptor) {
605
                    $jobDescriptor->write();
606
                }
607
608
                if (!$broken) {
609
                    $job->afterComplete();
610
                    $jobDescriptor->cleanupJob();
611
                }
612
            } catch (Exception $e) {
613
                // okay, we'll just catch this exception for now
614
                SS_Log::log($e, SS_Log::ERR);
615
                $jobDescriptor->JobStatus =  QueuedJob::STATUS_BROKEN;
616
                $jobDescriptor->write();
617
                $broken = true;
618
            }
619
        }
620
621
        $errorHandler->clear();
622
623
        Config::unnest();
624
625
        // okay lets reset our user if we've got an original
626
        if ($runAsUser && $originalUser) {
627
            Session::clear("loggedInAs");
628
            if ($originalUser) {
629
                Session::set("loggedInAs", $originalUser->ID);
630
            }
631
        }
632
633
        return !$broken;
634
    }
635
636
    /**
637
     * Start timer
638
     */
639
    protected function markStarted()
640
    {
641
        if ($this->startedAt) {
642
            $this->startedAt = SS_Datetime::now()->Format('U');
0 ignored issues
show
Documentation Bug introduced by
It seems like \SS_Datetime::now()->Format('U') can also be of type string. However, the property $startedAt is declared as type integer. Maybe add an additional type check?

Our type inference engine has found a suspicous assignment of a value to a property. This check raises an issue when a value that can be of a mixed type is assigned to a property that is type hinted more strictly.

For example, imagine you have a variable $accountId that can either hold an Id object or false (if there is no account id yet). Your code now assigns that value to the id property of an instance of the Account class. This class holds a proper account, so the id value must no longer be false.

Either this assignment is in error or a type check should be added for that assignment.

class Id
{
    public $id;

    public function __construct($id)
    {
        $this->id = $id;
    }

}

class Account
{
    /** @var  Id $id */
    public $id;
}

$account_id = false;

if (starsAreRight()) {
    $account_id = new Id(42);
}

$account = new Account();
if ($account instanceof Id)
{
    $account->id = $account_id;
}
Loading history...
643
        }
644
    }
645
646
    /**
647
     * Is execution time too long?
648
     *
649
     * @return bool True if the script has passed the configured time_limit
650
     */
651
    protected function hasPassedTimeLimit()
652
    {
653
        // Ensure a limit exists
654
        $limit = Config::inst()->get(__CLASS__, 'time_limit');
655
        if (!$limit) {
656
            return false;
657
        }
658
659
        // Ensure started date is set
660
        $this->markStarted();
661
662
        // Check duration
663
        $now = SS_Datetime::now()->Format('U');
664
        return $now > $this->startedAt + $limit;
665
    }
666
667
    /**
668
     * Is memory usage too high?
669
     *
670
     * @return bool
671
     */
672
    protected function isMemoryTooHigh()
673
    {
674
        $used = $this->getMemoryUsage();
675
        $limit = $this->getMemoryLimit();
676
        return $limit && ($used > $limit);
677
    }
678
679
    /**
680
     * Get peak memory usage of this application
681
     *
682
     * @return float
683
     */
684
    protected function getMemoryUsage()
685
    {
686
        // Note we use real_usage = false http://stackoverflow.com/questions/15745385/memory-get-peak-usage-with-real-usage
687
        // Also we use the safer peak memory usage
688
        return (float)memory_get_peak_usage(false);
689
    }
690
691
    /**
692
     * Determines the memory limit (in bytes) for this application
693
     * Limits to the smaller of memory_limit configured via php.ini or silverstripe config
694
     *
695
     * @return float Memory limit in bytes
696
     */
697
    protected function getMemoryLimit()
698
    {
699
        // Limit to smaller of explicit limit or php memory limit
700
        $limit = $this->parseMemory(Config::inst()->get(__CLASS__, 'memory_limit'));
701
        if ($limit) {
702
            return $limit;
703
        }
704
705
        // Fallback to php memory limit
706
        $phpLimit = $this->getPHPMemoryLimit();
707
        if ($phpLimit) {
708
            return $phpLimit;
709
        }
710
    }
711
712
    /**
713
     * Calculate the current memory limit of the server
714
     *
715
     * @return float
716
     */
717
    protected function getPHPMemoryLimit()
718
    {
719
        return $this->parseMemory(trim(ini_get("memory_limit")));
720
    }
721
722
    /**
723
     * Convert memory limit string to bytes.
724
     * Based on implementation in install.php5
725
     *
726
     * @param string $memString
727
     * @return float
728
     */
729
    protected function parseMemory($memString)
730
    {
731
        switch (strtolower(substr($memString, -1))) {
732
            case "b":
733
                return round(substr($memString, 0, -1));
734
            case "k":
735
                return round(substr($memString, 0, -1) * 1024);
736
            case "m":
737
                return round(substr($memString, 0, -1) * 1024 * 1024);
738
            case "g":
739
                return round(substr($memString, 0, -1) * 1024 * 1024 * 1024);
740
            default:
741
                return round($memString);
742
        }
743
    }
744
745
    protected function humanReadable($size)
746
    {
747
        $filesizename = array(" Bytes", " KB", " MB", " GB", " TB", " PB", " EB", " ZB", " YB");
748
        return $size ? round($size/pow(1024, ($i = floor(log($size, 1024)))), 2) . $filesizename[$i] : '0 Bytes';
749
    }
750
751
752
    /**
753
     * Gets a list of all the current jobs (or jobs that have recently finished)
754
     *
755
     * @param string $type
756
     *			if we're after a particular job list
757
     * @param int $includeUpUntil
758
     *			The number of seconds to include jobs that have just finished, allowing a job list to be built that
759
     *			includes recently finished jobs
760
     */
761
    public function getJobList($type = null, $includeUpUntil = 0)
762
    {
763
        return DataObject::get('QueuedJobDescriptor', $this->getJobListFilter($type, $includeUpUntil));
764
    }
765
766
    /**
767
     * Return the SQL filter used to get the job list - this is used by the UI for displaying the job list...
768
     *
769
     * @param string $type
770
     *			if we're after a particular job list
771
     * @param int $includeUpUntil
772
     *			The number of seconds to include jobs that have just finished, allowing a job list to be built that
773
     *			includes recently finished jobs
774
     * @return string
775
     */
776
    public function getJobListFilter($type = null, $includeUpUntil = 0)
777
    {
778
        $filter = array('JobStatus <>' => QueuedJob::STATUS_COMPLETE);
779
        if ($includeUpUntil) {
780
            $filter['JobFinished > '] = date('Y-m-d H:i:s', time() - $includeUpUntil);
781
        }
782
783
        $filter = singleton('QJUtils')->dbQuote($filter, ' OR ');
784
785
        if ($type) {
0 ignored issues
show
Bug Best Practice introduced by
The expression $type of type string|null is loosely compared to true; this is ambiguous if the string can be empty. You might want to explicitly use !== null instead.

In PHP, under loose comparison (like ==, or !=, or switch conditions), values of different types might be equal.

For string values, the empty string '' is a special case, in particular the following results might be unexpected:

''   == false // true
''   == null  // true
'ab' == false // false
'ab' == null  // false

// It is often better to use strict comparison
'' === false // false
'' === null  // false
Loading history...
786
            $filter = singleton('QJUtils')->dbQuote(array('JobType =' => (string) $type)) . ' AND ('.$filter.')';
787
        }
788
789
        return $filter;
790
    }
791
792
    /**
793
     * Process the job queue with the current queue runner
794
     *
795
     * @param string $queue
796
     */
797
    public function runQueue($queue)
798
    {
799
        $this->checkJobHealth();
800
        $this->queueRunner->runQueue($queue);
0 ignored issues
show
Bug introduced by
The property queueRunner does not seem to exist. Did you mean queuedRunner?

An attempt at access to an undefined property has been detected. This may either be a typographical error or the property has been renamed but there are still references to its old name.

If you really want to allow access to undefined properties, you can define magic methods to allow access. See the php core documentation on Overloading.

Loading history...
801
    }
802
803
    /**
804
     * Process all jobs from a given queue
805
     *
806
     * @param string $name The job queue to completely process
807
     */
808
    public function processJobQueue($name)
0 ignored issues
show
Coding Style introduced by
processJobQueue uses the super-global variable $_SESSION which is generally not recommended.

Instead of super-globals, we recommend to explicitly inject the dependencies of your class. This makes your code less dependent on global state and it becomes generally more testable:

// Bad
class Router
{
    public function generate($path)
    {
        return $_SERVER['HOST'].$path;
    }
}

// Better
class Router
{
    private $host;

    public function __construct($host)
    {
        $this->host = $host;
    }

    public function generate($path)
    {
        return $this->host.$path;
    }
}

class Controller
{
    public function myAction(Request $request)
    {
        // Instead of
        $page = isset($_GET['page']) ? intval($_GET['page']) : 1;

        // Better (assuming you use the Symfony2 request)
        $page = $request->query->get('page', 1);
    }
}
Loading history...
809
    {
810
        // Start timer to measure lifetime
811
        $this->markStarted();
812
813
        // Begin main loop
814
        do {
815
            if (class_exists('Subsite')) {
816
                // clear subsite back to default to prevent any subsite changes from leaking to
817
                // subsequent actions
818
                Subsite::changeSubsite(0);
819
            }
820
            if (Controller::has_curr()) {
821
                Session::clear('loggedInAs');
822
            } else {
823
                unset($_SESSION['loggedInAs']);
824
            }
825
826
            if (class_exists('SecurityContext')) {
827
                singleton('SecurityContext')->setMember(null);
828
            }
829
830
            $job = $this->getNextPendingJob($name);
831
            if ($job) {
832
                $success = $this->runJob($job->ID);
833
                if (!$success) {
834
                    // make sure job is null so it doesn't continue the current
835
                    // processing loop. Next queue executor can pick up where
836
                    // things left off
837
                    $job = null;
838
                }
839
            }
840
        } while ($job);
841
    }
842
843
    /**
844
     * When PHP shuts down, we want to process all of the immediate queue items
845
     *
846
     * We use the 'getNextPendingJob' method, instead of just iterating the queue, to ensure
847
     * we ignore paused or stalled jobs.
848
     */
849
    public function onShutdown()
850
    {
851
        $this->processJobQueue(QueuedJob::IMMEDIATE);
852
    }
853
}
854
855
/**
856
 * Class used to handle errors for a single job
857
 */
858
class JobErrorHandler
0 ignored issues
show
Coding Style Compatibility introduced by
PSR1 recommends that each class should be in its own file to aid autoloaders.

Having each class in a dedicated file usually plays nice with PSR autoloaders and is therefore a well established practice. If you use other autoloaders, you might not want to follow this rule.

Loading history...
Coding Style Compatibility introduced by
PSR1 recommends that each class must be in a namespace of at least one level to avoid collisions.

You can fix this by adding a namespace to your class:

namespace YourVendor;

class YourClass { }

When choosing a vendor namespace, try to pick something that is not too generic to avoid conflicts with other libraries.

Loading history...
859
{
860
    public function __construct()
861
    {
862
        set_error_handler(array($this, 'handleError'));
863
    }
864
865
    public function clear()
866
    {
867
        restore_error_handler();
868
    }
869
870
    public function handleError($errno, $errstr, $errfile, $errline)
871
    {
872
        if (error_reporting()) {
873
            // Don't throw E_DEPRECATED in PHP 5.3+
874
            if (defined('E_DEPRECATED')) {
875
                if ($errno == E_DEPRECATED || $errno = E_USER_DEPRECATED) {
876
                    return;
877
                }
878
            }
879
880
            switch ($errno) {
881
                case E_NOTICE:
882
                case E_USER_NOTICE:
883
                case E_STRICT: {
0 ignored issues
show
Coding Style introduced by
case statements should not use curly braces.

As per the PSR-2 coding standard, case statements should not be wrapped in curly braces. There is no need for braces, since each case is terminated by the next break.

switch ($expr) {
    case "A": { //wrong
        doSomething();
        break;
    }
    case "B": //right
        doSomething();
        break;
}

To learn more about the PSR-2 coding standard, please refer to the PHP-Fig.

Loading history...
884
                    break;
885
                }
886
                default: {
0 ignored issues
show
Coding Style introduced by
As per PSR2, default statements should not use curly braces.

As per the PSR-2 coding standard, default statements should not be wrapped in curly braces.

switch ($expr) {
    default: { //wrong
        doSomething();
        break;
    }
}

switch ($expr) {
    default: //right
        doSomething();
        break;
}

To learn more about the PSR-2 coding standard, please refer to the PHP-Fig.

Loading history...
887
                    throw new Exception($errstr . " in $errfile at line $errline", $errno);
888
                    break;
0 ignored issues
show
Unused Code introduced by
break; does not seem to be reachable.

This check looks for unreachable code. It uses sophisticated control flow analysis techniques to find statements which will never be executed.

Unreachable code is most often the result of return, die or exit statements that have been added for debug purposes.

function fx() {
    try {
        doSomething();
        return true;
    }
    catch (\Exception $e) {
        return false;
    }

    return false;
}

In the above example, the last return false will never be executed, because a return statement has already been met in every possible execution path.

Loading history...
889
                }
890
            }
891
        }
892
    }
893
}
894