Text::sanitizeFieldData() - Code Metrics - Inspection of "# Requesting a pull to nystudio107:v3 from timkelt..." - nystudio107/craft-seomatic - Measure and Improve Code Quality continuously with Scrutinizer

Passed

Pull Request — v3 (#685)

by Timothy

created 2020-07-14 12:59 UTC

Text::sanitizeFieldData() A

↳ Parent: Project

Complexity

Conditions	3
Paths	2

Size

Total Lines	17
Code Lines	10

Duplication

Lines	0
Ratio	0 %

Code Coverage

Tests	11
CRAP Score	3

Importance

Changes	2
Bugs	0	Features	0

Metric	Value
cc	3
eloc	10
nc	2
nop	1
dl	0
loc	17
ccs	11
cts	11
cp	1
crap	3
rs	9.9332
c	2
b	0
f	0

<?php
/**
 * SEOmatic plugin for Craft CMS 3.x
 *
 * A turnkey SEO implementation for Craft CMS that is comprehensive, powerful,
 * and flexible
 *
 * @link      https://nystudio107.com
 * @copyright Copyright (c) 2017 nystudio107
 */

namespace nystudio107\seomatic\helpers;

use nystudio107\seomatic\helpers\Field as FieldHelper;

use nystudio107\seomatic\Seomatic;

use craft\elements\db\MatrixBlockQuery;
use craft\elements\db\TagQuery;
use craft\elements\MatrixBlock;
use craft\elements\Tag;

use yii\base\InvalidConfigException;

use verbb\supertable\elements\SuperTableBlockElement as SuperTableBlock;
filter:
    dependency_paths: ["lib/*"]
use verbb\supertable\elements\db\SuperTableBlockQuery;
filter:
    dependency_paths: ["lib/*"]

use benf\neo\elements\db\BlockQuery as NeoBlockQuery;
filter:
    dependency_paths: ["lib/*"]
use benf\neo\elements\Block as NeoBlock;
filter:
    dependency_paths: ["lib/*"]

use Stringy\Stringy;

use PhpScience\TextRank\TextRankFacade;
use PhpScience\TextRank\Tool\StopWords\StopWordsAbstract;

/**
 * @author    nystudio107

 * @package   Seomatic
 * @since     3.0.0
 */
class Text
{
    // Constants
    // =========================================================================

    const LANGUAGE_MAP = [
        'en' => 'English',
        'fr' => 'French',
        'de' => 'German',
        'it' => 'Italian',
        'no' => 'Norwegian',
        'es' => 'Spanish',
    ];

    // Public Static Methods
    // =========================================================================

    /**
     * Truncates the string to a given length. If $substring is provided, and
     * truncating occurs, the string is further truncated so that the substring
     * may be appended without exceeding the desired length.
     *
     * @param  string $string    The string to truncate
     * @param  int    $length    Desired length of the truncated string
     * @param  string $substring The substring to append if it can fit
     *
     * @return string with the resulting $str after truncating
     */
    public static function truncate($string, $length, $substring = '…'): string
    {
        $result = $string;

        if (!empty($string)) {
            $string = strip_tags($string);
            $result = (string)Stringy::create($string)->truncate($length, $substring);
        }

        return $result;
    }

    /**
     * Truncates the string to a given length, while ensuring that it does not
     * split words. If $substring is provided, and truncating occurs, the
     * string is further truncated so that the substring may be appended without
     * exceeding the desired length.
     *
     * @param  string $string    The string to truncate
     * @param  int    $length    Desired length of the truncated string
     * @param  string $substring The substring to append if it can fit
     *
     * @return string with the resulting $str after truncating
     */
    public static function truncateOnWord($string, $length, $substring = '…'): string
    {
        $result = $string;

        if (!empty($string)) {
            $string = strip_tags($string);
            $result = (string)Stringy::create($string)->safeTruncate($length, $substring);
        }

        return $result;
    }

    /**
     * Extract plain old text from a field
     *
     * @param $field

     *
     * @return string
     */
    public static function extractTextFromField($field): string
    {
        if (empty($field)) {
            return '';
        }
        if ($field instanceof MatrixBlockQuery
            || (\is_array($field) && $field[0] instanceof MatrixBlock)) {

            $result = self::extractTextFromMatrix($field);
        } elseif ($field instanceof NeoBlockQuery
            || (\is_array($field) && $field[0] instanceof NeoBlock)) {

            $result = self::extractTextFromNeo($field);
        } elseif ($field instanceof SuperTableBlockQuery
            || (\is_array($field) && $field[0] instanceof SuperTableBlock)) {

            $result = self::extractTextFromSuperTable($field);
        } elseif ($field instanceof TagQuery
            || (\is_array($field) && $field[0] instanceof Tag)) {

            $result = self::extractTextFromTags($field);
        } else {
            if (\is_array($field)) {
                $result = self::smartStripTags((string)$field[0]);
            } else {
                $result = self::smartStripTags((string)$field);
            }
        }

        //return $result;
        return self::sanitizeFieldData($result);
    }

    /**
     * Extract concatenated text from all of the tags in the $tagElement and
     * return as a comma-delimited string
     *
     * @param TagQuery|Tag[] $tags

     *
     * @return string
     */
    public static function extractTextFromTags($tags): string
    {
        if (empty($tags)) {
            return '';
        }
        $result = '';
        // Iterate through all of the matrix blocks
        if ($tags instanceof TagQuery) {
            $tags = $tags->all();
        }
        foreach ($tags as $tag) {
            $result .= $tag->title.', ';
        }
        $result = rtrim($result, ', ');

        return $result;
    }

    /**
     * Extract text from all of the blocks in a matrix field, concatenating it
     * together.
     *
     * @param MatrixBlockQuery|MatrixBlock[] $blocks

     * @param string                         $fieldHandle

     *
     * @return string
     */
    public static function extractTextFromMatrix($blocks, $fieldHandle = ''): string
    {
        if (empty($blocks)) {
            return '';
        }
        $result = '';
        // Iterate through all of the matrix blocks
        if ($blocks instanceof MatrixBlockQuery) {
            $blocks = $blocks->all();
        }
        foreach ($blocks as $block) {
            try {
                $matrixBlockTypeModel = $block->getType();
            } catch (InvalidConfigException $e) {
                $matrixBlockTypeModel = null;
            }
            // Find any text fields inside of the matrix block
            if ($matrixBlockTypeModel) {
                $fieldClasses = FieldHelper::FIELD_CLASSES[FieldHelper::TEXT_FIELD_CLASS_KEY];
                $fields = $matrixBlockTypeModel->getFields();

                foreach ($fields as $field) {
                    /** @var array $fieldClasses */

                    foreach ($fieldClasses as $fieldClassKey) {
                        if ($field instanceof $fieldClassKey) {
                            if ($field->handle === $fieldHandle || empty($fieldHandle)) {
                                $result .= self::extractTextFromField($block[$field->handle]).' ';
                            }
                        }
                    }
                }
            }
        }

        return $result;
    }

    /**
     * Extract text from all of the blocks in a Neo field, concatenating it
     * together.
     *
     * @param NeoBlockQuery|NeoBlock[] $blocks

     * @param string                         $fieldHandle

     *
     * @return string
     */
    public static function extractTextFromNeo($blocks, $fieldHandle = ''): string
    {
        if (empty($blocks)) {
            return '';
        }
        $result = '';
        // Iterate through all of the matrix blocks
        if ($blocks instanceof NeoBlockQuery) {
            $blocks = $blocks->all();
        }
        foreach ($blocks as $block) {
            try {
                $neoBlockTypeModel = $block->getType();
            } catch (InvalidConfigException $e) {
                $neoBlockTypeModel = null;
            }
            // Find any text fields inside of the matrix block
            if ($neoBlockTypeModel) {
                $fieldClasses = FieldHelper::FIELD_CLASSES[FieldHelper::TEXT_FIELD_CLASS_KEY];
                $fields = $neoBlockTypeModel->getFields();

                foreach ($fields as $field) {
                    /** @var array $fieldClasses */

                    foreach ($fieldClasses as $fieldClassKey) {
                        if ($field instanceof $fieldClassKey) {
                            if ($field->handle === $fieldHandle || empty($fieldHandle)) {
                                $result .= self::extractTextFromField($block[$field->handle]).' ';
                            }
                        }
                    }
                }
            }
        }

        return $result;
    }

    /**
     * Extract text from all of the blocks in a matrix field, concatenating it
     * together.
     *
     * @param SuperTableBlockQuery|SuperTableBlock[] $blocks

     * @param string                         $fieldHandle

     *
     * @return string
     */
    public static function extractTextFromSuperTable($blocks, $fieldHandle = ''): string
    {
        if (empty($blocks)) {
            return '';
        }
        $result = '';
        // Iterate through all of the matrix blocks
        if ($blocks instanceof SuperTableBlockQuery) {
            $blocks = $blocks->all();
        }
        foreach ($blocks as $block) {
            try {
                $superTableBlockTypeModel = $block->getType();
            } catch (InvalidConfigException $e) {
                $superTableBlockTypeModel = null;
            }
            // Find any text fields inside of the matrix block
            if ($superTableBlockTypeModel) {
                $fieldClasses = FieldHelper::FIELD_CLASSES[FieldHelper::TEXT_FIELD_CLASS_KEY];
                $fields = $superTableBlockTypeModel->getFields();

                foreach ($fields as $field) {
                    /** @var array $fieldClasses */

                    foreach ($fieldClasses as $fieldClassKey) {
                        if ($field instanceof $fieldClassKey) {
                            if ($field->handle === $fieldHandle || empty($fieldHandle)) {
                                $result .= self::extractTextFromField($block[$field->handle]).' ';
                            }
                        }
                    }
                }
            }
        }

        return $result;
    }

    /**
     * Return the most important keywords extracted from the text as a comma-
     * delimited string
     *
     * @param string $text

     * @param int    $limit

     * @param bool   $useStopWords

     *
     * @return string
     */
    public static function extractKeywords($text, $limit = 15, $useStopWords = true): string
    {
        if (empty($text)) {
            return '';
        }
        $api = new TextRankFacade();
        // Set the stop words that should be ignored
        if ($useStopWords) {
            $language = strtolower(substr(Seomatic::$language, 0, 2));
            $stopWords = self::stopWordsForLanguage($language);
            if ($stopWords !== null) {
                $api->setStopWords($stopWords);
            }
        }
        // Array of the most important keywords:
        $keywords = $api->getOnlyKeyWords(self::cleanupText($text));

        // If it's empty, just return the text
        if (empty($keywords)) {
            return $text;
        }

        $result = \is_array($keywords)

            ? implode(', ', \array_slice(array_keys($keywords), 0, $limit))
            : (string)$keywords;

        return self::sanitizeFieldData($result);
    }

    /**
     * Extract a summary consisting of the 3 most important sentences from the
     * text
     *
     * @param string $text

     * @param bool   $useStopWords

     *
     * @return string
     */
    public static function extractSummary($text, $useStopWords = true): string
    {
        if (empty($text)) {
            return '';
        }
        $api = new TextRankFacade();
        // Set the stop words that should be ignored
        if ($useStopWords) {
            $language = strtolower(substr(Seomatic::$language, 0, 2));
            $stopWords = self::stopWordsForLanguage($language);
            if ($stopWords !== null) {
                $api->setStopWords($stopWords);
            }
        }
        // Array of the most important keywords:
        $sentences = $api->getHighlights(self::cleanupText($text));

        // If it's empty, just return the text
        if (empty($sentences)) {
            return $text;
        }

        $result = \is_array($sentences)

            ? implode(' ', $sentences)
            : (string)$sentences;

        return self::sanitizeFieldData($result);
    }


    /**
     * Sanitize Twig code out of any extracted field values
     *
     * @param $str

     * @return string

     */
    public static function sanitizeFieldData($str): string
    {
        // Do some general cleanup
        $str = html_entity_decode($str, ENT_NOQUOTES, 'UTF-8');
        $str = urldecode($str);
        $str = strip_tags($str);
        // Remove any embedded Twig code
        $str = preg_replace('/{{.*?}}/', '', $str);
        $str = preg_replace('/{%.*?%}/', '', $str);
        // Change single brackets to parenthesis
        $str = preg_replace('/{/', '(', $str);
        $str = preg_replace('/}/', ')', $str);
        if (empty($str) || is_array($str)) {
            $str = '';
        }

        return $str;
    }

    /**
     * Strip HTML tags, but replace them with a space rather than just eliminating them
     *
     * @param $str

     * @return string

     */
    public static function smartStripTags($str)
    {
        $str = str_replace('<', ' <', $str);
        $str = strip_tags($str);
        $str = str_replace('  ', ' ', $str);

        return $str;
    }

    /**
     * Clean up the passed in text by converting it to UTF-8, stripping tags,
     * removing whitespace, and decoding HTML entities
     *
     * @param string $text

     *
     * @return string
     */
    public static function cleanupText($text): string
    {
        if (empty($text)) {
            return '';
        }
        // Convert to UTF-8
        if (\function_exists('iconv')) {
            $text = iconv(mb_detect_encoding($text, mb_detect_order(), true), 'UTF-8//IGNORE', $text);
        } else {
            ini_set('mbstring.substitute_character', 'none');
            $text = mb_convert_encoding($text, 'UTF-8', 'UTF-8');
        }
        // Strip HTML tags
        $text = strip_tags($text);
        // Remove excess whitespace
        $text = preg_replace('/\s{2,}/u', ' ', $text);
        // Decode any HTML entities
        $text = html_entity_decode($text);

        return $text;
    }

    // Protected Static Methods
    // =========================================================================

    /**
     * @param string $language

     *
     * @return null|StopWordsAbstract
     */
    protected static function stopWordsForLanguage(string $language)
    {
        $stopWords = null;
        if (!empty(self::LANGUAGE_MAP[$language])) {
            $language = self::LANGUAGE_MAP[$language];
        } else {
            $language = 'English';
        }

        $className = 'PhpScience\\TextRank\\Tool\\StopWords\\'.ucfirst($language);
        if (class_exists($className)) {
            $stopWords = new $className;
        }

        return $stopWords;
    }
}


1			<?php
2			/**
3			* SEOmatic plugin for Craft CMS 3.x
4			*
5			* A turnkey SEO implementation for Craft CMS that is comprehensive, powerful,
6			* and flexible
7			*
8			* @link https://nystudio107.com
9			* @copyright Copyright (c) 2017 nystudio107
10			*/
11
12			namespace nystudio107\seomatic\helpers;
13
14			use nystudio107\seomatic\helpers\Field as FieldHelper;
15
16			use nystudio107\seomatic\Seomatic;
17
18			use craft\elements\db\MatrixBlockQuery;
19			use craft\elements\db\TagQuery;
20			use craft\elements\MatrixBlock;
21			use craft\elements\Tag;
22
23			use yii\base\InvalidConfigException;
24
25			use verbb\supertable\elements\SuperTableBlockElement as SuperTableBlock;
			0 ignored issues – show Bug introduced 2020-01-29 02:20 UTC by Report Bug Copy Issue Report The type `verbb\supertable\elements\SuperTableBlockElement` was not found. Maybe you did not declare it correctly or list all dependencies? The issue could also be caused by a filter entry in the build configuration. If the path has been excluded in your configuration, e.g. `excluded_paths: ["lib/"]`, you can move it to the dependency path list as follows: filter: dependency_paths: ["lib/"] For further information see https://scrutinizer-ci.com/docs/tools/php/php-scrutinizer/#list-dependency-paths Loading history...
26			use verbb\supertable\elements\db\SuperTableBlockQuery;
			0 ignored issues – show Bug introduced 2020-01-29 02:20 UTC by Report Bug Copy Issue Report The type `verbb\supertable\elements\db\SuperTableBlockQuery` was not found. Maybe you did not declare it correctly or list all dependencies? The issue could also be caused by a filter entry in the build configuration. If the path has been excluded in your configuration, e.g. `excluded_paths: ["lib/"]`, you can move it to the dependency path list as follows: filter: dependency_paths: ["lib/"] For further information see https://scrutinizer-ci.com/docs/tools/php/php-scrutinizer/#list-dependency-paths Loading history...
27
28			use benf\neo\elements\db\BlockQuery as NeoBlockQuery;
			0 ignored issues – show Bug introduced 2019-01-24 17:32 UTC by Report Bug Copy Issue Report The type `benf\neo\elements\db\BlockQuery` was not found. Maybe you did not declare it correctly or list all dependencies? The issue could also be caused by a filter entry in the build configuration. If the path has been excluded in your configuration, e.g. `excluded_paths: ["lib/"]`, you can move it to the dependency path list as follows: filter: dependency_paths: ["lib/"] For further information see https://scrutinizer-ci.com/docs/tools/php/php-scrutinizer/#list-dependency-paths Loading history...
29			use benf\neo\elements\Block as NeoBlock;
			0 ignored issues – show Bug introduced 2019-01-24 17:32 UTC by Report Bug Copy Issue Report The type `benf\neo\elements\Block` was not found. Maybe you did not declare it correctly or list all dependencies? The issue could also be caused by a filter entry in the build configuration. If the path has been excluded in your configuration, e.g. `excluded_paths: ["lib/"]`, you can move it to the dependency path list as follows: filter: dependency_paths: ["lib/"] For further information see https://scrutinizer-ci.com/docs/tools/php/php-scrutinizer/#list-dependency-paths Loading history...
30
31			use Stringy\Stringy;
32
33			use PhpScience\TextRank\TextRankFacade;
34			use PhpScience\TextRank\Tool\StopWords\StopWordsAbstract;
35
36			/**

nystudio107 / craft-seomatic

Pull Request — v3 (#685)

Text::sanitizeFieldData() A

Complexity

Size

Duplication

Code Coverage

Importance

Duplication Side-by-Side

Filter issues like