LostFilesCommand::execute() - Code Metrics - Inspection of "[TASK] Fix phpstan checkFunctionArgumentTypes erro..." - TYPO3/TYPO3.CMS - Measure and Improve Code Quality continuously with Scrutinizer

Completed

Push — master ( 1c797c...9045f9 )

unknown

created 2020-10-06 11:18 UTC

LostFilesCommand::execute() C

↳ Parent: LostFilesCommand

Complexity

Conditions	12
Paths	144

Size

Total Lines	46
Code Lines	26

Duplication

Lines	0
Ratio	0 %

Importance

Changes

Metric	Value
cc	12
eloc	26
c	0
b	0
f	0
nc	144
nop	2
dl	0
loc	46
rs	6.6

How to fix Complexity

<?php

declare(strict_types=1);

/*
 * This file is part of the TYPO3 CMS project.
 *
 * It is free software; you can redistribute it and/or modify it under
 * the terms of the GNU General Public License, either version 2
 * of the License, or any later version.
 *
 * For the full copyright and license information, please read the
 * LICENSE.txt file that was distributed with this source code.
 *
 * The TYPO3 project - inspiring people to share!
 */

namespace TYPO3\CMS\Lowlevel\Command;

use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Output\OutputInterface;
use Symfony\Component\Console\Style\SymfonyStyle;
use TYPO3\CMS\Backend\Command\ProgressListener\ReferenceIndexProgressListener;
use TYPO3\CMS\Core\Core\Bootstrap;
use TYPO3\CMS\Core\Core\Environment;
use TYPO3\CMS\Core\Database\ConnectionPool;
use TYPO3\CMS\Core\Database\ReferenceIndex;
use TYPO3\CMS\Core\Utility\GeneralUtility;

/**
 * Finds files within uploads/ which are not needed anymore
 */
class LostFilesCommand extends Command
{
    /**
     * @var ConnectionPool
     */
    private $connectionPool;

    public function __construct(ConnectionPool $connectionPool)
    {
        $this->connectionPool = $connectionPool;
        parent::__construct();
    }
    /**
     * Configure the command by defining the name, options and arguments
     */
    public function configure()
    {
        $this
            ->setDescription('Looking for files in the uploads/ folder which does not have a reference in TYPO3 managed records.')
            ->setHelp('
Assumptions:
- a perfect integrity of the reference index table (always update the reference index table before using this tool!)
- that all contents in the uploads folder are files attached to TCA records and exclusively managed by DataHandler through "group" type fields
- index.html, .htaccess files (ignored)
- Files found in deleted records are included (otherwise you would see a false list of lost files)

The assumptions are not requirements by the TYPO3 API but reflect the de facto implementation of most TYPO3 installations and therefore are a practical approach to clean up the uploads/ or custom folder.
Therefore, if all "group" type fields in TCA and flexforms are positioned inside the uploads/ folder and if no files inside are managed manually it should be safe to clean out files with no relations found in the system.
Under such circumstances there should theoretically be no lost files in the uploads/ or custom folder since DataHandler should have managed relations automatically including adding and deleting files.
However, there is at least one reason known to why files might be found lost and that is when FlexForms are used. In such a case a change of/in the Data Structure XML (or the ability of the system to find the Data Structure definition!) used for the flexform could leave lost files behind. This is not unlikely to happen when records are deleted. More details can be found in a note to the function FlexFormTools->getDataStructureIdentifier()
Another scenario could of course be de-installation of extensions which managed files in the uploads/ or custom folders.

If the option "--dry-run" is not set, the files are then deleted automatically.
Warning: First, make sure those files are not used somewhere TYPO3 does not know about! See the assumptions above.

If you want to get more detailed information, use the --verbose option.')
            ->addOption(
                'exclude',
                null,
                InputOption::VALUE_REQUIRED,
                'Comma-separated list of paths that should be excluded, e.g. "uploads/pics,uploads/media"'
            )
            ->addOption(
                'dry-run',
                null,
                InputOption::VALUE_NONE,
                'If this option is set, the files will not actually be deleted, but just the output which files would be deleted are shown'
            )
            ->addOption(
                'update-refindex',
                null,
                InputOption::VALUE_NONE,
                'Setting this option automatically updates the reference index and does not ask on command line. Alternatively, use -n to avoid the interactive mode'
            )
            ->addOption(
                'custom-path',
                null,
                InputOption::VALUE_REQUIRED,
                'Comma separated list of paths to process. Example: "fileadmin/[path1],fileadmin/[path2],...", if not passed, uploads/ will be used by default.'
            );
    }

    /**
     * Executes the command to
     * - optionally update the reference index (to have clean data)
     * - find files within uploads/* which are not connected to the reference index
     * - remove these files if --dry-run is not set
     *
     * @param InputInterface $input
     * @param OutputInterface $output
     * @return int
     */
    protected function execute(InputInterface $input, OutputInterface $output)
    {
        // Make sure the _cli_ user is loaded
        Bootstrap::initializeBackendAuthentication();

        $io = new SymfonyStyle($input, $output);
        $io->title($this->getDescription());

        $dryRun = $input->hasOption('dry-run') && $input->getOption('dry-run') != false ? true : false;

        $this->updateReferenceIndex($input, $io);

        // Find the lost files
        if ($input->hasOption('exclude') && !empty($input->getOption('exclude'))) {
            $exclude = $input->getOption('exclude');
            $exclude = is_string($exclude) ? $exclude : '';
            $excludedPaths = GeneralUtility::trimExplode(',', $exclude, true);
        } else {
            $excludedPaths = [];
        }

        // Use custom-path
        $customPaths = '';
        if ($input->hasOption('custom-path') && !empty($input->getOption('custom-path'))) {
            $customPaths = $input->getOption('custom-path');
            $customPaths = is_string($customPaths) ? $customPaths : '';
        }

        $lostFiles = $this->findLostFiles($excludedPaths, $customPaths);

        if (count($lostFiles)) {
            if (!$io->isQuiet()) {
                $io->note('Found ' . count($lostFiles) . ' lost files, ready to be deleted.');
                if ($io->isVerbose()) {
                    $io->listing($lostFiles);
                }
            }

            // Delete them
            $this->deleteLostFiles($lostFiles, $dryRun, $io);

            $io->success('Deleted ' . count($lostFiles) . ' lost files.');
        } else {
            $io->success('Nothing to do, no lost files found');
        }
        return 0;
    }

    /**
     * Function to update the reference index
     * - if the option --update-refindex is set, do it
     * - otherwise, if in interactive mode (not having -n set), ask the user
     * - otherwise assume everything is fine
     *
     * @param InputInterface $input holds information about entered parameters
     * @param SymfonyStyle $io necessary for outputting information
     */
    protected function updateReferenceIndex(InputInterface $input, SymfonyStyle $io)
    {
        // Check for reference index to update
        $io->note('Finding lost files managed by TYPO3 requires a clean reference index (sys_refindex)');
        $updateReferenceIndex = false;
        if ($input->hasOption('update-refindex') && $input->getOption('update-refindex')) {
            $updateReferenceIndex = true;
        } elseif ($input->isInteractive()) {
            $updateReferenceIndex = $io->confirm('Should the reference index be updated right now?', false);
        }

        // Update the reference index
        if ($updateReferenceIndex) {
            $progressListener = GeneralUtility::makeInstance(ReferenceIndexProgressListener::class);
            $progressListener->initialize($io);
            $referenceIndex = GeneralUtility::makeInstance(ReferenceIndex::class);
            $io->section('Reference Index is now being updated');
            $referenceIndex->updateIndex(false, $progressListener);
        } else {
            $io->writeln('Reference index is assumed to be up to date, continuing.');
        }
    }

    /**
     * Find lost files in uploads/ or custom folder
     *
     * @param array $excludedPaths list of paths to be excluded, can be uploads/pics/
     * @param string $customPaths list of paths to be checked instead of uploads/
     * @return array an array of files (relative to Environment::getPublicPath()) that are not connected
     */
    protected function findLostFiles($excludedPaths = [], $customPaths = ''): array
    {
        $lostFiles = [];

        // Get all files
        $files = [];
        if (!empty($customPaths)) {
            $customPaths = GeneralUtility::trimExplode(',', $customPaths, true);
            foreach ($customPaths as $customPath) {
                if (false === realpath(Environment::getPublicPath() . '/' . $customPath)
                    || !GeneralUtility::isFirstPartOfStr((string)realpath(Environment::getPublicPath() . '/' . $customPath), (string)realpath(Environment::getPublicPath()))) {
                    throw new \Exception('The path: "' . $customPath . '" is invalid', 1450086736);
                }
                $files = GeneralUtility::getAllFilesAndFoldersInPath($files, Environment::getPublicPath() . '/' . $customPath);
            }
        } else {
            $files = GeneralUtility::getAllFilesAndFoldersInPath($files, Environment::getPublicPath() . '/uploads/');
        }

        $files = GeneralUtility::removePrefixPathFromList($files, Environment::getPublicPath() . '/');

        $queryBuilder = $this->connectionPool
            ->getQueryBuilderForTable('sys_refindex');

        // Traverse files and for each, look up if its found in the reference index.
        foreach ($files as $key => $value) {

            // First, allow "index.html", ".htaccess" files since they are often used for good reasons
            if (substr($value, -11) === '/index.html' || substr($value, -10) === '/.htaccess') {
                continue;
            }

            $fileIsInExcludedPath = false;
            foreach ($excludedPaths as $exclPath) {
                if (GeneralUtility::isFirstPartOfStr($value, $exclPath)) {
                    $fileIsInExcludedPath = true;
                    break;
                }
            }

            if ($fileIsInExcludedPath) {
                continue;
            }

            // Looking for a reference from a field which is NOT a soft reference (thus, only fields with a proper TCA/Flexform configuration)
            $queryBuilder
                ->select('hash')
                ->from('sys_refindex')
                ->where(
                    $queryBuilder->expr()->eq(
                        'ref_table',
                        $queryBuilder->createNamedParameter('_FILE', \PDO::PARAM_STR)
                    ),
                    $queryBuilder->expr()->eq(
                        'ref_string',
                        $queryBuilder->createNamedParameter($value, \PDO::PARAM_STR)
                    ),
                    $queryBuilder->expr()->eq(
                        'softref_key',
                        $queryBuilder->createNamedParameter('', \PDO::PARAM_STR)
                    )
                )
                ->orderBy('sorting', 'DESC')
                ->execute();

            $rowCount = $queryBuilder->count('hash')->execute()->fetchColumn(0);
            // We conclude that the file is lost
            if ($rowCount === 0) {
                $lostFiles[] = $value;
            }
        }

        return $lostFiles;
    }

    /**
     * Removes given files from the uploads/ folder
     *
     * @param array $lostFiles Contains the lost files found
     * @param bool $dryRun if set, the files are just displayed, but not deleted
     * @param SymfonyStyle $io the IO object for output
     */
    protected function deleteLostFiles(array $lostFiles, bool $dryRun, SymfonyStyle $io)
    {
        foreach ($lostFiles as $lostFile) {
            $absoluteFileName = GeneralUtility::getFileAbsFileName($lostFile);
            if ($io->isVeryVerbose()) {
                $io->writeln('Deleting file "' . $absoluteFileName . '"');
            }
            if (!$dryRun) {
                if ($absoluteFileName && @is_file($absoluteFileName)) {
                    unlink($absoluteFileName);
                    if (!$io->isQuiet()) {
                        $io->writeln('Permanently deleted file record "' . $absoluteFileName . '".');
                    }
                } else {
                    $io->error('File "' . $absoluteFileName . '" was not found!');
                }
            }
        }
    }
}


1			<?php
2
3			declare(strict_types=1);
4
5			/*
6			* This file is part of the TYPO3 CMS project.
7			*
8			* It is free software; you can redistribute it and/or modify it under
9			* the terms of the GNU General Public License, either version 2
10			* of the License, or any later version.
11			*
12			* For the full copyright and license information, please read the
13			* LICENSE.txt file that was distributed with this source code.
14			*
15			* The TYPO3 project - inspiring people to share!
16			*/
17
18			namespace TYPO3\CMS\Lowlevel\Command;
19
20			use Symfony\Component\Console\Command\Command;
21			use Symfony\Component\Console\Input\InputInterface;
22			use Symfony\Component\Console\Input\InputOption;
23			use Symfony\Component\Console\Output\OutputInterface;
24			use Symfony\Component\Console\Style\SymfonyStyle;
25			use TYPO3\CMS\Backend\Command\ProgressListener\ReferenceIndexProgressListener;
26			use TYPO3\CMS\Core\Core\Bootstrap;
27			use TYPO3\CMS\Core\Core\Environment;
28			use TYPO3\CMS\Core\Database\ConnectionPool;
29			use TYPO3\CMS\Core\Database\ReferenceIndex;
30			use TYPO3\CMS\Core\Utility\GeneralUtility;
31
32			/**
33			* Finds files within uploads/ which are not needed anymore
34			*/
35			class LostFilesCommand extends Command
36			{
37			/**
38			* @var ConnectionPool
39			*/
40			private $connectionPool;
41
42			public function __construct(ConnectionPool $connectionPool)
43			{
44			$this->connectionPool = $connectionPool;
45			parent::__construct();
46			}
47			/**
48			* Configure the command by defining the name, options and arguments
49			*/
50			public function configure()
51			{
52			$this
53			->setDescription('Looking for files in the uploads/ folder which does not have a reference in TYPO3 managed records.')
54			->setHelp('
55			Assumptions:
56			- a perfect integrity of the reference index table (always update the reference index table before using this tool!)
57			- that all contents in the uploads folder are files attached to TCA records and exclusively managed by DataHandler through "group" type fields
58			- index.html, .htaccess files (ignored)
59			- Files found in deleted records are included (otherwise you would see a false list of lost files)
60
61			The assumptions are not requirements by the TYPO3 API but reflect the de facto implementation of most TYPO3 installations and therefore are a practical approach to clean up the uploads/ or custom folder.
62			Therefore, if all "group" type fields in TCA and flexforms are positioned inside the uploads/ folder and if no files inside are managed manually it should be safe to clean out files with no relations found in the system.
63			Under such circumstances there should theoretically be no lost files in the uploads/ or custom folder since DataHandler should have managed relations automatically including adding and deleting files.
64			However, there is at least one reason known to why files might be found lost and that is when FlexForms are used. In such a case a change of/in the Data Structure XML (or the ability of the system to find the Data Structure definition!) used for the flexform could leave lost files behind. This is not unlikely to happen when records are deleted. More details can be found in a note to the function FlexFormTools->getDataStructureIdentifier()
65			Another scenario could of course be de-installation of extensions which managed files in the uploads/ or custom folders.
66
67			If the option "--dry-run" is not set, the files are then deleted automatically.
68			Warning: First, make sure those files are not used somewhere TYPO3 does not know about! See the assumptions above.
69
70			If you want to get more detailed information, use the --verbose option.')
71			->addOption(
72			'exclude',
73			null,
74			InputOption::VALUE_REQUIRED,
75			'Comma-separated list of paths that should be excluded, e.g. "uploads/pics,uploads/media"'
76			)
77			->addOption(
78			'dry-run',
79			null,
80			InputOption::VALUE_NONE,
81			'If this option is set, the files will not actually be deleted, but just the output which files would be deleted are shown'
82			)
83			->addOption(
84			'update-refindex',
85			null,
86			InputOption::VALUE_NONE,
87			'Setting this option automatically updates the reference index and does not ask on command line. Alternatively, use -n to avoid the interactive mode'
88			)
89			->addOption(
90			'custom-path',
91			null,
92			InputOption::VALUE_REQUIRED,
93			'Comma separated list of paths to process. Example: "fileadmin/[path1],fileadmin/[path2],...", if not passed, uploads/ will be used by default.'
94			);
95			}
96
97			/**
98			* Executes the command to
99			* - optionally update the reference index (to have clean data)
100			* - find files within uploads/* which are not connected to the reference index
101			* - remove these files if --dry-run is not set
102			*
103			* @param InputInterface $input
104			* @param OutputInterface $output
105			* @return int
106			*/
107			protected function execute(InputInterface $input, OutputInterface $output)
108			{
109			// Make sure the _cli_ user is loaded
110			Bootstrap::initializeBackendAuthentication();
111
112			$io = new SymfonyStyle($input, $output);
113			$io->title($this->getDescription());
114
115			$dryRun = $input->hasOption('dry-run') && $input->getOption('dry-run') != false ? true : false;
116
117			$this->updateReferenceIndex($input, $io);
118
119			// Find the lost files
120			if ($input->hasOption('exclude') && !empty($input->getOption('exclude'))) {
121			$exclude = $input->getOption('exclude');
122			$exclude = is_string($exclude) ? $exclude : '';
123			$excludedPaths = GeneralUtility::trimExplode(',', $exclude, true);
124			} else {
125			$excludedPaths = [];
126			}
127
128			// Use custom-path
129			$customPaths = '';
130			if ($input->hasOption('custom-path') && !empty($input->getOption('custom-path'))) {
131			$customPaths = $input->getOption('custom-path');
132			$customPaths = is_string($customPaths) ? $customPaths : '';
133			}
134
135			$lostFiles = $this->findLostFiles($excludedPaths, $customPaths);
136
137			if (count($lostFiles)) {
138			if (!$io->isQuiet()) {
139			$io->note('Found ' . count($lostFiles) . ' lost files, ready to be deleted.');
140			if ($io->isVerbose()) {
141			$io->listing($lostFiles);
142			}
143			}
144
145			// Delete them
146			$this->deleteLostFiles($lostFiles, $dryRun, $io);
147
148			$io->success('Deleted ' . count($lostFiles) . ' lost files.');
149			} else {
150			$io->success('Nothing to do, no lost files found');
151			}
152			return 0;
153			}
154
155			/**
156			* Function to update the reference index
157			* - if the option --update-refindex is set, do it
158			* - otherwise, if in interactive mode (not having -n set), ask the user
159			* - otherwise assume everything is fine
160			*
161			* @param InputInterface $input holds information about entered parameters
162			* @param SymfonyStyle $io necessary for outputting information
163			*/
164			protected function updateReferenceIndex(InputInterface $input, SymfonyStyle $io)
165			{
166			// Check for reference index to update
167			$io->note('Finding lost files managed by TYPO3 requires a clean reference index (sys_refindex)');
168			$updateReferenceIndex = false;
169			if ($input->hasOption('update-refindex') && $input->getOption('update-refindex')) {
170			$updateReferenceIndex = true;
171			} elseif ($input->isInteractive()) {
172			$updateReferenceIndex = $io->confirm('Should the reference index be updated right now?', false);
173			}
174
175			// Update the reference index
176			if ($updateReferenceIndex) {
177			$progressListener = GeneralUtility::makeInstance(ReferenceIndexProgressListener::class);
178			$progressListener->initialize($io);
179			$referenceIndex = GeneralUtility::makeInstance(ReferenceIndex::class);
180			$io->section('Reference Index is now being updated');
181			$referenceIndex->updateIndex(false, $progressListener);
182			} else {
183			$io->writeln('Reference index is assumed to be up to date, continuing.');
184			}
185			}
186
187			/**
188			* Find lost files in uploads/ or custom folder
189			*
190			* @param array $excludedPaths list of paths to be excluded, can be uploads/pics/
191			* @param string $customPaths list of paths to be checked instead of uploads/
192			* @return array an array of files (relative to Environment::getPublicPath()) that are not connected
193			*/
194			protected function findLostFiles($excludedPaths = [], $customPaths = ''): array
195			{
196			$lostFiles = [];
197
198			// Get all files
199			$files = [];
200			if (!empty($customPaths)) {
201			$customPaths = GeneralUtility::trimExplode(',', $customPaths, true);
202			foreach ($customPaths as $customPath) {
203			if (false === realpath(Environment::getPublicPath() . '/' . $customPath)
204			\|\| !GeneralUtility::isFirstPartOfStr((string)realpath(Environment::getPublicPath() . '/' . $customPath), (string)realpath(Environment::getPublicPath()))) {
205			throw new \Exception('The path: "' . $customPath . '" is invalid', 1450086736);
206			}
207			$files = GeneralUtility::getAllFilesAndFoldersInPath($files, Environment::getPublicPath() . '/' . $customPath);
208			}
209			} else {
210			$files = GeneralUtility::getAllFilesAndFoldersInPath($files, Environment::getPublicPath() . '/uploads/');
211			}
212
213			$files = GeneralUtility::removePrefixPathFromList($files, Environment::getPublicPath() . '/');
214
215			$queryBuilder = $this->connectionPool
216			->getQueryBuilderForTable('sys_refindex');
217
218			// Traverse files and for each, look up if its found in the reference index.
219			foreach ($files as $key => $value) {
220
221			// First, allow "index.html", ".htaccess" files since they are often used for good reasons
222			if (substr($value, -11) === '/index.html' \|\| substr($value, -10) === '/.htaccess') {
223			continue;
224			}
225
226			$fileIsInExcludedPath = false;
227			foreach ($excludedPaths as $exclPath) {
228			if (GeneralUtility::isFirstPartOfStr($value, $exclPath)) {
229			$fileIsInExcludedPath = true;
230			break;
231			}
232			}
233
234			if ($fileIsInExcludedPath) {
235			continue;
236			}
237
238			// Looking for a reference from a field which is NOT a soft reference (thus, only fields with a proper TCA/Flexform configuration)
239			$queryBuilder
240			->select('hash')
241			->from('sys_refindex')
242			->where(
243			$queryBuilder->expr()->eq(
244			'ref_table',
245			$queryBuilder->createNamedParameter('_FILE', \PDO::PARAM_STR)
246			),
247			$queryBuilder->expr()->eq(
248			'ref_string',
249			$queryBuilder->createNamedParameter($value, \PDO::PARAM_STR)
250			),
251			$queryBuilder->expr()->eq(
252			'softref_key',
253			$queryBuilder->createNamedParameter('', \PDO::PARAM_STR)
254			)
255			)
256			->orderBy('sorting', 'DESC')
257			->execute();
258
259			$rowCount = $queryBuilder->count('hash')->execute()->fetchColumn(0);
260			// We conclude that the file is lost
261			if ($rowCount === 0) {
262			$lostFiles[] = $value;
263			}
264			}
265
266			return $lostFiles;
267			}
268
269			/**
270			* Removes given files from the uploads/ folder
271			*
272			* @param array $lostFiles Contains the lost files found
273			* @param bool $dryRun if set, the files are just displayed, but not deleted
274			* @param SymfonyStyle $io the IO object for output
275			*/
276			protected function deleteLostFiles(array $lostFiles, bool $dryRun, SymfonyStyle $io)
277			{
278			foreach ($lostFiles as $lostFile) {
279			$absoluteFileName = GeneralUtility::getFileAbsFileName($lostFile);
280			if ($io->isVeryVerbose()) {
281			$io->writeln('Deleting file "' . $absoluteFileName . '"');
282			}
283			if (!$dryRun) {
284			if ($absoluteFileName && @is_file($absoluteFileName)) {
285			unlink($absoluteFileName);
286			if (!$io->isQuiet()) {
287			$io->writeln('Permanently deleted file record "' . $absoluteFileName . '".');
288			}
289			} else {
290			$io->error('File "' . $absoluteFileName . '" was not found!');
291			}
292			}
293			}
294			}
295			}
296

TYPO3 / TYPO3.CMS

Push — master ( 1c797c...9045f9 )

LostFilesCommand::execute() C

Complexity

Size

Duplication

Importance

How to fix Complexity

Long Method

Duplication Side-by-Side

Filter issues like