Issues in compressOld.php (master) - Issues in master - wikimedia/mediawiki - Measure and Improve Code Quality continuously with Scrutinizer

Issues (4122)

Security Analysis not enabled

Cross-Site Scripting

Cross-Site Scripting enables an attacker to inject code into the response of a web-request that is viewed by other users. It can for example be used to bypass access controls, or even to take over other users' accounts.

File Exposure

File Exposure allows an attacker to gain access to local files that he should not be able to access. These files can for example include database credentials, or other configuration files.

File Manipulation

File Manipulation enables an attacker to write custom data to files. This potentially leads to injection of arbitrary code on the server.

Object Injection

Object Injection enables an attacker to inject an object into PHP code, and can lead to arbitrary code execution, file exposure, or file manipulation attacks.

Code Injection

Code Injection enables an attacker to execute arbitrary code on the server.

Response Splitting

Response Splitting can be used to send arbitrary responses.

File Inclusion

File Inclusion enables an attacker to inject custom files into PHP's file loading mechanism, either explicitly passed to include, or for example via PHP's auto-loading mechanism.

Command Injection

Command Injection enables an attacker to inject a shell command that is execute with the privileges of the web-server. This can be used to expose sensitive data, or gain access of your server.

SQL Injection

SQL Injection enables an attacker to execute arbitrary SQL code on your database server gaining access to user data, or manipulating user data.

XPath Injection

XPath Injection enables an attacker to modify the parts of XML document that are read. If that XML document is for example used for authentication, this can lead to further vulnerabilities similar to SQL Injection.

LDAP Injection

LDAP Injection enables an attacker to inject LDAP statements potentially granting permission to run unauthorized queries, or modify content inside the LDAP tree.

Header Injection

Other Vulnerability

This category comprises other attack vectors such as manipulating the PHP runtime, loading custom extensions, freezing the runtime, or similar.

Regex Injection

Regex Injection enables an attacker to execute arbitrary code in your PHP process.

XML Injection

XML Injection enables an attacker to read files on your local filesystem including configuration files, or can be abused to freeze your web-server process.

Variable Injection

Variable Injection enables an attacker to overwrite program variables with custom data, and can lead to further vulnerabilities.

Unfortunately, the security analysis is currently not available for your project. If you are a non-commercial open-source project, please contact support to gain access.

maintenance/storage/compressOld.php (7 issues)

Labels

Severity

Upgrade to new PHP Analysis Engine

These results are based on our legacy PHP analysis, consider migrating to our new PHP analysis engine instead. Learn more

<?php

/**
 * Compress the text of a wiki.
 *
 * Usage:
 *
 * Non-wikimedia
 * php compressOld.php [options...]
 *
 * Wikimedia
 * php compressOld.php <database> [options...]
 *
 * Options are:
 *  -t <type>           set compression type to either:
 *                          gzip: compress revisions independently
 *                          concat: concatenate revisions and compress in chunks (default)
 *  -c <chunk-size>     maximum number of revisions in a concat chunk
 *  -b <begin-date>     earliest date to check for uncompressed revisions
 *  -e <end-date>       latest revision date to compress
 *  -s <startid>        the id to start from (referring to the text table for
 *                      type gzip, and to the page table for type concat)
 *  -n <endid>          the page_id to stop at (only when using concat compression type)
 *  --extdb <cluster>   store specified revisions in an external cluster (untested)
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation; either version 2 of the License, or
 * (at your option) any later version.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU General Public License along
 * with this program; if not, write to the Free Software Foundation, Inc.,
 * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
 * http://www.gnu.org/copyleft/gpl.html
 *
 * @file
 * @ingroup Maintenance ExternalStorage
 */

require_once __DIR__ . '/../Maintenance.php';

/**
 * Maintenance script that compress the text of a wiki.
 *
 * @ingroup Maintenance ExternalStorage
 */
class CompressOld extends Maintenance {
	/**
	 * Option to load each revision individually.
	 *
	 */
	const LS_INDIVIDUAL = 0;

	/**
	 * Option to load revisions in chunks.
	 *
	 */
	const LS_CHUNKED = 1;

	public function __construct() {
		parent::__construct();
		$this->addDescription( 'Compress the text of a wiki' );
		$this->addOption( 'type', 'Set compression type to either: gzip|concat', false, true, 't' );
		$this->addOption(
			'chunksize',
			'Maximum number of revisions in a concat chunk',
			false,
			true,
			'c'
		);
		$this->addOption(
			'begin-date',
			'Earliest date to check for uncompressed revisions',
			false,
			true,
			'b'
		);
		$this->addOption( 'end-date', 'Latest revision date to compress', false, true, 'e' );
		$this->addOption(
			'startid',
			'The id to start from (gzip -> text table, concat -> page table)',
			false,
			true,
			's'
		);
		$this->addOption(
			'extdb',
			'Store specified revisions in an external cluster (untested)',
			false,
			true
		);
		$this->addOption(
			'endid',
			'The page_id to stop at (only when using concat compression type)',
			false,
			true,
			'n'
		);
	}

	public function execute() {
		global $wgDBname;
		if ( !function_exists( "gzdeflate" ) ) {
			$this->error( "You must enable zlib support in PHP to compress old revisions!\n" .
				"Please see http://www.php.net/manual/en/ref.zlib.php\n", true );
		}

		$type = $this->getOption( 'type', 'concat' );
		$chunkSize = $this->getOption( 'chunksize', 20 );
		$startId = $this->getOption( 'startid', 0 );
		$beginDate = $this->getOption( 'begin-date', '' );
		$endDate = $this->getOption( 'end-date', '' );
		$extDB = $this->getOption( 'extdb', '' );
		$endId = $this->getOption( 'endid', false );

		if ( $type != 'concat' && $type != 'gzip' ) {
			$this->error( "Type \"{$type}\" not supported" );
		}

		if ( $extDB != '' ) {
			$this->output( "Compressing database {$wgDBname} to external cluster {$extDB}\n"
				. str_repeat( '-', 76 ) . "\n\n" );
		} else {
			$this->output( "Compressing database {$wgDBname}\n"
				. str_repeat( '-', 76 ) . "\n\n" );
		}

		$success = true;
		if ( $type == 'concat' ) {
			$success = $this->compressWithConcat( $startId, $chunkSize, $beginDate,
				$endDate, $extDB, $endId );
		} else {
			$this->compressOldPages( $startId, $extDB );
		}

		if ( $success ) {
			$this->output( "Done.\n" );
		}
	}

	/**
	 * Fetch the text row-by-row to 'compressPage' function for compression.
	 *
	 * @param int $start
	 * @param string $extdb
	 */
	private function compressOldPages( $start = 0, $extdb = '' ) {
		$chunksize = 50;
		$this->output( "Starting from old_id $start...\n" );
		$dbw = $this->getDB( DB_MASTER );
		do {
			$res = $dbw->select(
				'text',
				[ 'old_id', 'old_flags', 'old_text' ],
				"old_id>=$start",
				__METHOD__,
				[ 'ORDER BY' => 'old_id', 'LIMIT' => $chunksize, 'FOR UPDATE' ]
			);

			if ( $res->numRows() == 0 ) {
				break;
			}

			$last = $start;

			foreach ( $res as $row ) {
				# print "  {$row->old_id} - {$row->old_namespace}:{$row->old_title}\n";
				$this->compressPage( $row, $extdb );
				$last = $row->old_id;
			}

			$start = $last + 1; # Deletion may leave long empty stretches
			$this->output( "$start...\n" );
		} while ( true );
	}

	/**
	 * Compress the text in gzip format.
	 *
	 * @param stdClass $row
	 * @param string $extdb
	 * @return bool
	 */
	private function compressPage( $row, $extdb ) {
		if ( false !== strpos( $row->old_flags, 'gzip' )
			|| false !== strpos( $row->old_flags, 'object' )
		) {
			# print "Already compressed row {$row->old_id}\n";
			return false;
		}
		$dbw = $this->getDB( DB_MASTER );
		$flags = $row->old_flags ? "{$row->old_flags},gzip" : "gzip";
		$compress = gzdeflate( $row->old_text );

		# Store in external storage if required
		if ( $extdb !== '' ) {
			$storeObj = new ExternalStoreDB;
			$compress = $storeObj->store( $extdb, $compress );
			if ( $compress === false ) {
				$this->error( "Unable to store object" );

				return false;
			}
		}

		# Update text row
		$dbw->update( 'text',
			[ /* SET */
				'old_flags' => $flags,
				'old_text' => $compress
			], [ /* WHERE */
				'old_id' => $row->old_id
			], __METHOD__,
			[ 'LIMIT' => 1 ]
		);

		return true;
	}

	/**
	 * Compress the text in chunks after concatenating the revisions.
	 *
	 * @param int $startId
	 * @param int $maxChunkSize
	 * @param string $beginDate
	 * @param string $endDate
	 * @param string $extdb
	 * @param bool|int $maxPageId
	 * @return bool
	 */
	private function compressWithConcat( $startId, $maxChunkSize, $beginDate,
		$endDate, $extdb = "", $maxPageId = false
	) {
		$loadStyle = self::LS_CHUNKED;

		$dbr = $this->getDB( DB_REPLICA );
		$dbw = $this->getDB( DB_MASTER );

		# Set up external storage
		if ( $extdb != '' ) {
			$storeObj = new ExternalStoreDB;
		}

		# Get all articles by page_id
		if ( !$maxPageId ) {
			$maxPageId = $dbr->selectField( 'page', 'max(page_id)', '', __METHOD__ );
		}
		$this->output( "Starting from $startId of $maxPageId\n" );
		$pageConds = [];

		/*
		if ( $exclude_ns0 ) {
			print "Excluding main namespace\n";
			$pageConds[] = 'page_namespace<>0';
		}
		if ( $queryExtra ) {
					$pageConds[] = $queryExtra;
		}
		 */

		# For each article, get a list of revisions which fit the criteria

		# No recompression, use a condition on old_flags
		# Don't compress object type entities, because that might produce data loss when
		# overwriting bulk storage concat rows. Don't compress external references, because
		# the script doesn't yet delete rows from external storage.
		$conds = [
			'old_flags NOT ' . $dbr->buildLike( $dbr->anyString(), 'object', $dbr->anyString() )
			. ' AND old_flags NOT '
			. $dbr->buildLike( $dbr->anyString(), 'external', $dbr->anyString() )
		];

		if ( $beginDate ) {
			if ( !preg_match( '/^\d{14}$/', $beginDate ) ) {
				$this->error( "Invalid begin date \"$beginDate\"\n" );

				return false;
			}
			$conds[] = "rev_timestamp>'" . $beginDate . "'";
		}
		if ( $endDate ) {
			if ( !preg_match( '/^\d{14}$/', $endDate ) ) {
				$this->error( "Invalid end date \"$endDate\"\n" );

				return false;
			}
			$conds[] = "rev_timestamp<'" . $endDate . "'";
		}
		if ( $loadStyle == self::LS_CHUNKED ) {
			$tables = [ 'revision', 'text' ];
			$fields = [ 'rev_id', 'rev_text_id', 'old_flags', 'old_text' ];
			$conds[] = 'rev_text_id=old_id';
			$revLoadOptions = 'FOR UPDATE';
		} else {
			$tables = [ 'revision' ];
			$fields = [ 'rev_id', 'rev_text_id' ];
			$revLoadOptions = [];
		}

		# Don't work with current revisions
		# Don't lock the page table for update either -- TS 2006-04-04
		# $tables[] = 'page';
		# $conds[] = 'page_id=rev_page AND rev_id != page_latest';

		for ( $pageId = $startId; $pageId <= $maxPageId; $pageId++ ) {
			wfWaitForSlaves();


			# Wake up
			$dbr->ping();

			# Get the page row
			$pageRes = $dbr->select( 'page',
				[ 'page_id', 'page_namespace', 'page_title', 'page_latest' ],
				$pageConds + [ 'page_id' => $pageId ], __METHOD__ );
			if ( $pageRes->numRows() == 0 ) {
				continue;
			}
			$pageRow = $dbr->fetchObject( $pageRes );

			# Display progress
			$titleObj = Title::makeTitle( $pageRow->page_namespace, $pageRow->page_title );
			$this->output( "$pageId\t" . $titleObj->getPrefixedDBkey() . " " );

			# Load revisions
			$revRes = $dbw->select( $tables, $fields,
				array_merge( [
					'rev_page' => $pageRow->page_id,
					# Don't operate on the current revision
					# Use < instead of <> in case the current revision has changed
					# since the page select, which wasn't locking
					'rev_id < ' . $pageRow->page_latest
				], $conds ),
				__METHOD__,
				$revLoadOptions
/**
 * @return array|string
 */
function returnsDifferentValues($x) {
    if ($x) {
        return 'foo';
    }

    return array();
}

$x = returnsDifferentValues($y);
if (is_array($x)) {
    // $x is an array.
}
			);
			$revs = [];
			foreach ( $revRes as $revRow ) {
				$revs[] = $revRow;
			}

			if ( count( $revs ) < 2 ) {
				# No revisions matching, no further processing
				$this->output( "\n" );
				continue;
			}

			# For each chunk
			$i = 0;
			while ( $i < count( $revs ) ) {
				if ( $i < count( $revs ) - $maxChunkSize ) {
					$thisChunkSize = $maxChunkSize;
				} else {
					$thisChunkSize = count( $revs ) - $i;
				}

				$chunk = new ConcatenatedGzipHistoryBlob();
				$stubs = [];
				$this->beginTransaction( $dbw, __METHOD__ );
/** @return stdClass|null */
function mayReturnNull() { }

function doesNotAcceptNull(stdClass $x) { }

// With potential error.
function withoutCheck() {
    $x = mayReturnNull();
    doesNotAcceptNull($x); // Potential error here.
}

// Safe - Alternative 1
function withCheck1() {
    $x = mayReturnNull();
    if ( ! $x instanceof stdClass) {
        throw new \LogicException('$x must be defined.');
    }
    doesNotAcceptNull($x);
}

// Safe - Alternative 2
function withCheck2() {
    $x = mayReturnNull();
    if ($x instanceof stdClass) {
        doesNotAcceptNull($x);
    }
}
				$usedChunk = false;
				$primaryOldid = $revs[$i]->rev_text_id;

				// @codingStandardsIgnoreStart Ignore avoid function calls in a FOR loop test part warning
				# Get the text of each revision and add it to the object
				for ( $j = 0; $j < $thisChunkSize && $chunk->isHappy(); $j++ ) {
					// @codingStandardsIgnoreEnd
					$oldid = $revs[$i + $j]->rev_text_id;

					# Get text
					if ( $loadStyle == self::LS_INDIVIDUAL ) {
						$textRow = $dbw->selectRow( 'text',
							[ 'old_flags', 'old_text' ],
							[ 'old_id' => $oldid ],
							__METHOD__,
							'FOR UPDATE'
						);
						$text = Revision::getRevisionText( $textRow );
					} else {
						$text = Revision::getRevisionText( $revs[$i + $j] );
					}

					if ( $text === false ) {
						$this->error( "\nError, unable to get text in old_id $oldid" );
						# $dbw->delete( 'old', [ 'old_id' => $oldid ] );
					}

					if ( $extdb == "" && $j == 0 ) {
						$chunk->setText( $text );
						$this->output( '.' );
					} else {
						# Don't make a stub if it's going to be longer than the article
						# Stubs are typically about 100 bytes
						if ( strlen( $text ) < 120 ) {
							$stub = false;
							$this->output( 'x' );
						} else {
							$stub = new HistoryBlobStub( $chunk->addItem( $text ) );
							$stub->setLocation( $primaryOldid );
							$stub->setReferrer( $oldid );
							$this->output( '.' );
							$usedChunk = true;
						}
						$stubs[$j] = $stub;
					}
				}
				$thisChunkSize = $j;

				# If we couldn't actually use any stubs because the pages were too small, do nothing
				if ( $usedChunk ) {
					if ( $extdb != "" ) {
						# Move blob objects to External Storage
						$stored = $storeObj->store( $extdb, serialize( $chunk ) );
function myFunction($a) {
    switch ($a) {
        case 'foo':
            $x = 1;
            break;

        case 'bar':
            $x = 2;
            break;
    }

    // $x is potentially undefined here.
    echo $x;
}
						if ( $stored === false ) {
							$this->error( "Unable to store object" );

							return false;
						}
						# Store External Storage URLs instead of Stub placeholders
						foreach ( $stubs as $stub ) {
							if ( $stub === false ) {
								continue;
							}
							# $stored should provide base path to a BLOB
							$url = $stored . "/" . $stub->getHash();
							$dbw->update( 'text',
								[ /* SET */
									'old_text' => $url,
									'old_flags' => 'external,utf-8',
								], [ /* WHERE */
									'old_id' => $stub->getReferrer(),
								]
							);
						}
					} else {
						# Store the main object locally
						$dbw->update( 'text',
							[ /* SET */
								'old_text' => serialize( $chunk ),
								'old_flags' => 'object,utf-8',
							], [ /* WHERE */
								'old_id' => $primaryOldid
							]
						);

						# Store the stub objects
						for ( $j = 1; $j < $thisChunkSize; $j++ ) {
							# Skip if not compressing and don't overwrite the first revision
							if ( $stubs[$j] !== false && $revs[$i + $j]->rev_text_id != $primaryOldid ) {
								$dbw->update( 'text',
									[ /* SET */
										'old_text' => serialize( $stubs[$j] ),
										'old_flags' => 'object,utf-8',
									], [ /* WHERE */
										'old_id' => $revs[$i + $j]->rev_text_id
									]
								);
							}
						}
					}
				}
				# Done, next
				$this->output( "/" );
				$this->commitTransaction( $dbw, __METHOD__ );
/** @return stdClass|null */
function mayReturnNull() { }

function doesNotAcceptNull(stdClass $x) { }

// With potential error.
function withoutCheck() {
    $x = mayReturnNull();
    doesNotAcceptNull($x); // Potential error here.
}

// Safe - Alternative 1
function withCheck1() {
    $x = mayReturnNull();
    if ( ! $x instanceof stdClass) {
        throw new \LogicException('$x must be defined.');
    }
    doesNotAcceptNull($x);
}

// Safe - Alternative 2
function withCheck2() {
    $x = mayReturnNull();
    if ($x instanceof stdClass) {
        doesNotAcceptNull($x);
    }
}
				$i += $thisChunkSize;
				wfWaitForSlaves();

			}
			$this->output( "\n" );
		}

		return true;
	}
}

$maintClass = 'CompressOld';
require_once RUN_MAINTENANCE_IF_MAIN;


Issues (4122)

Security Analysis not enabled

maintenance/storage/compressOld.php (7 issues)

Labels

Severity

Introduced By

Upgrade to new PHP Analysis Engine

Available Fixes

1		<?php
		0 ignored issues – show Coding Style Compatibility introduced 2016-02-23 18:14 UTC by Report Bug Copy Issue Report Show Similar Issues like this For compatibility and reusability of your code, PSR1 recommends that a file should introduce either new symbols (like classes, functions, etc.) or have side-effects (like outputting something, or including other files), but not both at the same time. The first symbol is defined on line 51 and the first side effect is on line 44. The PSR-1: Basic Coding Standard recommends that a file should either introduce new symbols, that is classes, functions, constants or similar, or have side effects. Side effects are anything that executes logic, like for example printing output, changing ini settings or writing to a file. The idea behind this recommendation is that merely auto-loading a class should not change the state of an application. It also promotes a cleaner style of programming and makes your code less prone to errors, because the logic is not spread out all over the place. To learn more about the PSR-1, please see the PHP-FIG site on the PSR-1. Loading history...
2		/**
3		* Compress the text of a wiki.
4		*
5		* Usage:
6		*
7		* Non-wikimedia
8		* php compressOld.php [options...]
9		*
10		* Wikimedia
11		* php compressOld.php <database> [options...]
12		*
13		* Options are:
14		* -t <type> set compression type to either:
15		* gzip: compress revisions independently
16		* concat: concatenate revisions and compress in chunks (default)
17		* -c <chunk-size> maximum number of revisions in a concat chunk
18		* -b <begin-date> earliest date to check for uncompressed revisions
19		* -e <end-date> latest revision date to compress
20		* -s <startid> the id to start from (referring to the text table for
21		* type gzip, and to the page table for type concat)
22		* -n <endid> the page_id to stop at (only when using concat compression type)
23		* --extdb <cluster> store specified revisions in an external cluster (untested)
24		*
25		* This program is free software; you can redistribute it and/or modify
26		* it under the terms of the GNU General Public License as published by
27		* the Free Software Foundation; either version 2 of the License, or
28		* (at your option) any later version.
29		*
30		* This program is distributed in the hope that it will be useful,
31		* but WITHOUT ANY WARRANTY; without even the implied warranty of
32		* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
33		* GNU General Public License for more details.
34		*
35		* You should have received a copy of the GNU General Public License along
36		* with this program; if not, write to the Free Software Foundation, Inc.,
37		* 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
38		* http://www.gnu.org/copyleft/gpl.html
39		*
40		* @file
41		* @ingroup Maintenance ExternalStorage
42		*/
43
44		require_once __DIR__ . '/../Maintenance.php';
45
46		/**
47		* Maintenance script that compress the text of a wiki.
48		*
49		* @ingroup Maintenance ExternalStorage
50		*/
51		class CompressOld extends Maintenance {
52		/**
53		* Option to load each revision individually.
54		*
55		*/
56		const LS_INDIVIDUAL = 0;
57
58		/**
59		* Option to load revisions in chunks.
60		*
61		*/
62		const LS_CHUNKED = 1;
63
64		public function __construct() {
65		parent::__construct();
66		$this->addDescription( 'Compress the text of a wiki' );
67		$this->addOption( 'type', 'Set compression type to either: gzip\|concat', false, true, 't' );
68		$this->addOption(
69		'chunksize',
70		'Maximum number of revisions in a concat chunk',
71		false,
72		true,
73		'c'
74		);
75		$this->addOption(
76		'begin-date',
77		'Earliest date to check for uncompressed revisions',
78		false,
79		true,
80		'b'
81		);
82		$this->addOption( 'end-date', 'Latest revision date to compress', false, true, 'e' );
83		$this->addOption(
84		'startid',
85		'The id to start from (gzip -> text table, concat -> page table)',
86		false,
87		true,
88		's'
89		);
90		$this->addOption(
91		'extdb',
92		'Store specified revisions in an external cluster (untested)',
93		false,
94		true
95		);
96		$this->addOption(
97		'endid',
98		'The page_id to stop at (only when using concat compression type)',
99		false,
100		true,
101		'n'
102		);
103		}
104
105		public function execute() {
106		global $wgDBname;
107		if ( !function_exists( "gzdeflate" ) ) {
108		$this->error( "You must enable zlib support in PHP to compress old revisions!\n" .
109		"Please see http://www.php.net/manual/en/ref.zlib.php\n", true );
110		}
111
112		$type = $this->getOption( 'type', 'concat' );
113		$chunkSize = $this->getOption( 'chunksize', 20 );
114		$startId = $this->getOption( 'startid', 0 );
115		$beginDate = $this->getOption( 'begin-date', '' );
116		$endDate = $this->getOption( 'end-date', '' );
117		$extDB = $this->getOption( 'extdb', '' );
118		$endId = $this->getOption( 'endid', false );
119
120		if ( $type != 'concat' && $type != 'gzip' ) {
121		$this->error( "Type \"{$type}\" not supported" );
122		}
123
124		if ( $extDB != '' ) {
125		$this->output( "Compressing database {$wgDBname} to external cluster {$extDB}\n"
126		. str_repeat( '-', 76 ) . "\n\n" );
127		} else {
128		$this->output( "Compressing database {$wgDBname}\n"
129		. str_repeat( '-', 76 ) . "\n\n" );
130		}
131
132		$success = true;
133		if ( $type == 'concat' ) {
134		$success = $this->compressWithConcat( $startId, $chunkSize, $beginDate,
135		$endDate, $extDB, $endId );
136		} else {
137		$this->compressOldPages( $startId, $extDB );
138		}
139
140		if ( $success ) {
141		$this->output( "Done.\n" );
142		}
143		}
144
145		/**
146		* Fetch the text row-by-row to 'compressPage' function for compression.
147		*
148		* @param int $start
149		* @param string $extdb
150		*/
151		private function compressOldPages( $start = 0, $extdb = '' ) {
152		$chunksize = 50;
153		$this->output( "Starting from old_id $start...\n" );
154		$dbw = $this->getDB( DB_MASTER );
155		do {
156		$res = $dbw->select(
157		'text',
158		[ 'old_id', 'old_flags', 'old_text' ],
159		"old_id>=$start",
160		__METHOD__,
161		[ 'ORDER BY' => 'old_id', 'LIMIT' => $chunksize, 'FOR UPDATE' ]
162		);
163
164		if ( $res->numRows() == 0 ) {
165		break;
166		}
167
168		$last = $start;
169
170		foreach ( $res as $row ) {
171		# print " {$row->old_id} - {$row->old_namespace}:{$row->old_title}\n";
172		$this->compressPage( $row, $extdb );
173		$last = $row->old_id;
174		}
175
176		$start = $last + 1; # Deletion may leave long empty stretches
177		$this->output( "$start...\n" );
178		} while ( true );
179		}
180
181		/**
182		* Compress the text in gzip format.
183		*
184		* @param stdClass $row
185		* @param string $extdb
186		* @return bool
187		*/
188		private function compressPage( $row, $extdb ) {
189		if ( false !== strpos( $row->old_flags, 'gzip' )
190		\|\| false !== strpos( $row->old_flags, 'object' )
191		) {
192		# print "Already compressed row {$row->old_id}\n";
193		return false;
194		}
195		$dbw = $this->getDB( DB_MASTER );
196		$flags = $row->old_flags ? "{$row->old_flags},gzip" : "gzip";
197		$compress = gzdeflate( $row->old_text );
198
199		# Store in external storage if required
200		if ( $extdb !== '' ) {
201		$storeObj = new ExternalStoreDB;
202		$compress = $storeObj->store( $extdb, $compress );
203		if ( $compress === false ) {
204		$this->error( "Unable to store object" );
205
206		return false;
207		}
208		}
209
210		# Update text row
211		$dbw->update( 'text',
212		[ /* SET */
213		'old_flags' => $flags,
214		'old_text' => $compress
215		], [ /* WHERE */
216		'old_id' => $row->old_id
217		], __METHOD__,
218		[ 'LIMIT' => 1 ]
219		);
220
221		return true;
222		}
223
224		/**
225		* Compress the text in chunks after concatenating the revisions.
226		*
227		* @param int $startId
228		* @param int $maxChunkSize
229		* @param string $beginDate
230		* @param string $endDate
231		* @param string $extdb
232		* @param bool\|int $maxPageId
233		* @return bool
234		*/
235		private function compressWithConcat( $startId, $maxChunkSize, $beginDate,
236		$endDate, $extdb = "", $maxPageId = false
237		) {
238		$loadStyle = self::LS_CHUNKED;
239
240		$dbr = $this->getDB( DB_REPLICA );
241		$dbw = $this->getDB( DB_MASTER );
242
243		# Set up external storage
244		if ( $extdb != '' ) {
245		$storeObj = new ExternalStoreDB;
246		}
247
248		# Get all articles by page_id
249		if ( !$maxPageId ) {
250		$maxPageId = $dbr->selectField( 'page', 'max(page_id)', '', __METHOD__ );
251		}
252		$this->output( "Starting from $startId of $maxPageId\n" );
253		$pageConds = [];
254
255		/*
256		if ( $exclude_ns0 ) {
257		print "Excluding main namespace\n";
258		$pageConds[] = 'page_namespace<>0';
259		}
260		if ( $queryExtra ) {
261		$pageConds[] = $queryExtra;
262		}
263		*/
264
265		# For each article, get a list of revisions which fit the criteria
266
267		# No recompression, use a condition on old_flags
268		# Don't compress object type entities, because that might produce data loss when
269		# overwriting bulk storage concat rows. Don't compress external references, because
270		# the script doesn't yet delete rows from external storage.
271		$conds = [
272		'old_flags NOT ' . $dbr->buildLike( $dbr->anyString(), 'object', $dbr->anyString() )
273		. ' AND old_flags NOT '
274		. $dbr->buildLike( $dbr->anyString(), 'external', $dbr->anyString() )
275		];
276
277	View Code Duplication	if ( $beginDate ) {
278		if ( !preg_match( '/^\d{14}$/', $beginDate ) ) {
279		$this->error( "Invalid begin date \"$beginDate\"\n" );
280
281		return false;
282		}
283		$conds[] = "rev_timestamp>'" . $beginDate . "'";
284		}
285	View Code Duplication	if ( $endDate ) {
286		if ( !preg_match( '/^\d{14}$/', $endDate ) ) {
287		$this->error( "Invalid end date \"$endDate\"\n" );
288
289		return false;
290		}
291		$conds[] = "rev_timestamp<'" . $endDate . "'";
292		}
293		if ( $loadStyle == self::LS_CHUNKED ) {
294		$tables = [ 'revision', 'text' ];
295		$fields = [ 'rev_id', 'rev_text_id', 'old_flags', 'old_text' ];
296		$conds[] = 'rev_text_id=old_id';
297		$revLoadOptions = 'FOR UPDATE';
298		} else {
299		$tables = [ 'revision' ];
300		$fields = [ 'rev_id', 'rev_text_id' ];
301		$revLoadOptions = [];
302		}
303
304		# Don't work with current revisions
305		# Don't lock the page table for update either -- TS 2006-04-04
306		# $tables[] = 'page';
307		# $conds[] = 'page_id=rev_page AND rev_id != page_latest';
308
309		for ( $pageId = $startId; $pageId <= $maxPageId; $pageId++ ) {
310		wfWaitForSlaves();
		0 ignored issues – show Deprecated Code introduced 2016-01-25 18:14 UTC by Report Bug Copy Issue Report Show Similar Issues like this The function `wfWaitForSlaves()` has been deprecated with message: since 1.27 Use LBFactory::waitForReplication This function has been deprecated. The supplier of the file has supplied an explanatory message. The explanatory message should give you some clue as to whether and when the function will be removed from the class and what other function to use instead. Loading history...
311
312		# Wake up
313		$dbr->ping();
314
315		# Get the page row
316		$pageRes = $dbr->select( 'page',
317		[ 'page_id', 'page_namespace', 'page_title', 'page_latest' ],
318		$pageConds + [ 'page_id' => $pageId ], __METHOD__ );
319		if ( $pageRes->numRows() == 0 ) {
320		continue;
321		}
322		$pageRow = $dbr->fetchObject( $pageRes );
323
324		# Display progress
325		$titleObj = Title::makeTitle( $pageRow->page_namespace, $pageRow->page_title );
326		$this->output( "$pageId\t" . $titleObj->getPrefixedDBkey() . " " );
327
328		# Load revisions
329		$revRes = $dbw->select( $tables, $fields,
330		array_merge( [
331		'rev_page' => $pageRow->page_id,
332		# Don't operate on the current revision
333		# Use < instead of <> in case the current revision has changed
334		# since the page select, which wasn't locking
335		'rev_id < ' . $pageRow->page_latest
336		], $conds ),
337		__METHOD__,
338		$revLoadOptions
		0 ignored issues – show Bug introduced 2016-09-23 18:28 UTC by Report Bug Copy Issue Report Show Similar Issues like this It seems like `$revLoadOptions` defined by `'FOR UPDATE'` on line `297` can also be of type `string`; however, `Database::select()` does only seem to accept `array`, maybe add an additional type check? If a method or function can return multiple different values and unless you are sure that you only can receive a single value in this context, we recommend to add an additional type check: /** * @return array\|string */ function returnsDifferentValues($x) { if ($x) { return 'foo'; } return array(); } $x = returnsDifferentValues($y); if (is_array($x)) { // $x is an array. } If this a common case that PHP Analyzer should handle natively, please let us know by opening an issue. Loading history...
339		);
340		$revs = [];
341		foreach ( $revRes as $revRow ) {
342		$revs[] = $revRow;
343		}
344
345		if ( count( $revs ) < 2 ) {
346		# No revisions matching, no further processing
347		$this->output( "\n" );
348		continue;
349		}
350
351		# For each chunk
352		$i = 0;
353		while ( $i < count( $revs ) ) {
354		if ( $i < count( $revs ) - $maxChunkSize ) {
355		$thisChunkSize = $maxChunkSize;
356		} else {
357		$thisChunkSize = count( $revs ) - $i;
358		}
359
360		$chunk = new ConcatenatedGzipHistoryBlob();
361		$stubs = [];
362		$this->beginTransaction( $dbw, __METHOD__ );
		0 ignored issues – show Bug introduced 2016-09-24 18:22 UTC by Report Bug Copy Issue Report Show Similar Issues like this It seems like `$dbw` defined by `$this->getDB(DB_MASTER)` on line `241` can be `null`; however, `Maintenance::beginTransaction()` does not accept `null`, maybe add an additional type check? Unless you are absolutely sure that the expression can never be null because of other conditions, we strongly recommend to add an additional type check to your code: /** @return stdClass\|null */ function mayReturnNull() { } function doesNotAcceptNull(stdClass $x) { } // With potential error. function withoutCheck() { $x = mayReturnNull(); doesNotAcceptNull($x); // Potential error here. } // Safe - Alternative 1 function withCheck1() { $x = mayReturnNull(); if ( ! $x instanceof stdClass) { throw new \LogicException('$x must be defined.'); } doesNotAcceptNull($x); } // Safe - Alternative 2 function withCheck2() { $x = mayReturnNull(); if ($x instanceof stdClass) { doesNotAcceptNull($x); } } Loading history...
363		$usedChunk = false;
364		$primaryOldid = $revs[$i]->rev_text_id;
365
366		// @codingStandardsIgnoreStart Ignore avoid function calls in a FOR loop test part warning
367		# Get the text of each revision and add it to the object
368		for ( $j = 0; $j < $thisChunkSize && $chunk->isHappy(); $j++ ) {
369		// @codingStandardsIgnoreEnd
370		$oldid = $revs[$i + $j]->rev_text_id;
371
372		# Get text
373		if ( $loadStyle == self::LS_INDIVIDUAL ) {
374		$textRow = $dbw->selectRow( 'text',
375		[ 'old_flags', 'old_text' ],
376		[ 'old_id' => $oldid ],
377		__METHOD__,
378		'FOR UPDATE'
379		);
380		$text = Revision::getRevisionText( $textRow );
381		} else {
382		$text = Revision::getRevisionText( $revs[$i + $j] );
383		}
384
385		if ( $text === false ) {
386		$this->error( "\nError, unable to get text in old_id $oldid" );
387		# $dbw->delete( 'old', [ 'old_id' => $oldid ] );
388		}
389
390		if ( $extdb == "" && $j == 0 ) {
391		$chunk->setText( $text );
392		$this->output( '.' );
393		} else {
394		# Don't make a stub if it's going to be longer than the article
395		# Stubs are typically about 100 bytes
396		if ( strlen( $text ) < 120 ) {
397		$stub = false;
398		$this->output( 'x' );
399		} else {
400		$stub = new HistoryBlobStub( $chunk->addItem( $text ) );
401		$stub->setLocation( $primaryOldid );
402		$stub->setReferrer( $oldid );
403		$this->output( '.' );
404		$usedChunk = true;
405		}
406		$stubs[$j] = $stub;
407		}
408		}
409		$thisChunkSize = $j;
410
411		# If we couldn't actually use any stubs because the pages were too small, do nothing
412		if ( $usedChunk ) {
413		if ( $extdb != "" ) {
414		# Move blob objects to External Storage
415		$stored = $storeObj->store( $extdb, serialize( $chunk ) );
		0 ignored issues – show Bug introduced 2016-01-16 18:00 UTC by Report Bug Copy Issue Report Show Similar Issues like this The variable `$storeObj` does not seem to be defined for all execution paths leading up to this point. If you define a variable conditionally, it can happen that it is not defined for all execution paths. Let’s take a look at an example: function myFunction($a) { switch ($a) { case 'foo': $x = 1; break; case 'bar': $x = 2; break; } // $x is potentially undefined here. echo $x; } In the above example, the variable `$x` is defined if you pass “foo” or “bar” as argument for `$a`. However, since the `switch` statement has no default case statement, if you pass any other value, the variable `$x` would be undefined. Available Fixes Check for existence of the variable explicitly: function myFunction($a) { switch ($a) { case 'foo': $x = 1; break; case 'bar': $x = 2; break; } if (isset($x)) { // Make sure it's always set. echo $x; } } Define a default value for the variable: function myFunction($a) { $x = ''; // Set a default which gets overridden for certain paths. switch ($a) { case 'foo': $x = 1; break; case 'bar': $x = 2; break; } echo $x; } Add a value for the missing path: function myFunction($a) { switch ($a) { case 'foo': $x = 1; break; case 'bar': $x = 2; break; // We add support for the missing case. default: $x = ''; break; } echo $x; } Loading history...
416		if ( $stored === false ) {
417		$this->error( "Unable to store object" );
418
419		return false;
420		}
421		# Store External Storage URLs instead of Stub placeholders
422		foreach ( $stubs as $stub ) {
423		if ( $stub === false ) {
424		continue;
425		}
426		# $stored should provide base path to a BLOB
427		$url = $stored . "/" . $stub->getHash();
428		$dbw->update( 'text',
429		[ /* SET */
430		'old_text' => $url,
431		'old_flags' => 'external,utf-8',
432		], [ /* WHERE */
433		'old_id' => $stub->getReferrer(),
434		]
435		);
436		}
437		} else {
438		# Store the main object locally
439		$dbw->update( 'text',
440		[ /* SET */
441		'old_text' => serialize( $chunk ),
442		'old_flags' => 'object,utf-8',
443		], [ /* WHERE */
444		'old_id' => $primaryOldid
445		]
446		);
447
448		# Store the stub objects
449		for ( $j = 1; $j < $thisChunkSize; $j++ ) {
450		# Skip if not compressing and don't overwrite the first revision
451		if ( $stubs[$j] !== false && $revs[$i + $j]->rev_text_id != $primaryOldid ) {
452		$dbw->update( 'text',
453		[ /* SET */
454		'old_text' => serialize( $stubs[$j] ),
455		'old_flags' => 'object,utf-8',
456		], [ /* WHERE */
457		'old_id' => $revs[$i + $j]->rev_text_id
458		]
459		);
460		}
461		}
462		}
463		}
464		# Done, next
465		$this->output( "/" );
466		$this->commitTransaction( $dbw, __METHOD__ );
		0 ignored issues – show Bug introduced 2016-09-24 18:22 UTC by Report Bug Copy Issue Report Show Similar Issues like this It seems like `$dbw` defined by `$this->getDB(DB_MASTER)` on line `241` can be `null`; however, `Maintenance::commitTransaction()` does not accept `null`, maybe add an additional type check? Unless you are absolutely sure that the expression can never be null because of other conditions, we strongly recommend to add an additional type check to your code: /** @return stdClass\|null */ function mayReturnNull() { } function doesNotAcceptNull(stdClass $x) { } // With potential error. function withoutCheck() { $x = mayReturnNull(); doesNotAcceptNull($x); // Potential error here. } // Safe - Alternative 1 function withCheck1() { $x = mayReturnNull(); if ( ! $x instanceof stdClass) { throw new \LogicException('$x must be defined.'); } doesNotAcceptNull($x); } // Safe - Alternative 2 function withCheck2() { $x = mayReturnNull(); if ($x instanceof stdClass) { doesNotAcceptNull($x); } } Loading history...
467		$i += $thisChunkSize;
468		wfWaitForSlaves();
		0 ignored issues – show Deprecated Code introduced 2016-01-25 18:14 UTC by Report Bug Copy Issue Report Show Similar Issues like this The function `wfWaitForSlaves()` has been deprecated with message: since 1.27 Use LBFactory::waitForReplication This function has been deprecated. The supplier of the file has supplied an explanatory message. The explanatory message should give you some clue as to whether and when the function will be removed from the class and what other function to use instead. Loading history...
469		}
470		$this->output( "\n" );
471		}
472
473		return true;
474		}
475		}
476
477		$maintClass = 'CompressOld';
478		require_once RUN_MAINTENANCE_IF_MAIN;
479

wikimedia / mediawiki

Issues (4122)

Security Analysis not enabled

maintenance/storage/compressOld.php (7 issues)

Labels

Severity

Introduced By

Upgrade to new PHP Analysis Engine

Available Fixes

Duplication Side-by-Side

Filter issues like