transliteration.php ➔ _transliteration_replace() - Code Metrics - Inspection of "Code cleanup (WIP)" - seblucas/cops - Measure and Improve Code Quality continuously with Scrutinizer

Completed

Pull Request — master (#274)

by Markus

created 2016-10-23 22:53 UTC

transliteration.php ➔ _transliteration_replace() B

↳ Parent: Project

Complexity

Conditions	6
Paths	8

Size

Total Lines	36
Code Lines	17

Duplication

Lines	0
Ratio	0 %

Code Coverage

Tests	14
CRAP Score	6.1979

Importance

Changes

Metric	Value
cc	6
eloc	17
nc	8
nop	3
dl	0
loc	36
ccs	14
cts	17
cp	0.8235
crap	6.1979
rs	8.439
c	0
b	0
f	0

<?php

/**
 * @file
 * Transliteration processing functions.
 */

/**
 * Transliterates UTF-8 encoded text to US-ASCII.
 *
 * Based on Mediawiki's UtfNormal::quickIsNFCVerify().
 * Based on Drupal 7 transliteration module.
 *
 * @param $string
 *   UTF-8 encoded text input.
 * @param $unknown
 *   Replacement string for characters that do not have a suitable ASCII
 *   equivalent.
 * @param $source_langcode
 *   Optional ISO 639 language code that denotes the language of the input and
 *   is used to apply language-specific variations. If the source language is
 *   not known at the time of transliteration, it is recommended to set this
 *   argument to the site default language to produce consistent results.
 *   Otherwise the current display language will be used.
 * @return
 *   Transliterated text.
 */
function _transliteration_process($string, $unknown = '?', $source_langcode = NULL) {

  // ASCII is always valid NFC! If we're only ever given plain ASCII, we can
  // avoid the overhead of initializing the decomposition tables by skipping
  // out early.
  if (!preg_match('/[\x80-\xff]/', $string)) {
    return $string;
  }

  static $tail_bytes;

  if (!isset($tail_bytes)) {
    // Each UTF-8 head byte is followed by a certain number of tail bytes.
    $tail_bytes = array();
    for ($n = 0; $n < 256; $n++) {
      if ($n < 0xc0) {
        $remaining = 0;
      }
      elseif ($n < 0xe0) {
        $remaining = 1;
      }
      elseif ($n < 0xf0) {
        $remaining = 2;
      }
      elseif ($n < 0xf8) {
        $remaining = 3;
      }
      elseif ($n < 0xfc) {
        $remaining = 4;
      }
      elseif ($n < 0xfe) {
        $remaining = 5;
      }
      else {
        $remaining = 0;
      }
      $tail_bytes[chr($n)] = $remaining;
    }
  }

  // Chop the text into pure-ASCII and non-ASCII areas; large ASCII parts can
  // be handled much more quickly. Don't chop up Unicode areas for punctuation,
  // though, that wastes energy.
  preg_match_all('/[\x00-\x7f]+|[\x80-\xff][\x00-\x40\x5b-\x5f\x7b-\xff]*/', $string, $matches);

  $result = '';
  foreach ($matches[0] as $str) {
    if ($str[0] < "\x80") {
      // ASCII chunk: guaranteed to be valid UTF-8 and in normal form C, so
      // skip over it.
      $result .= $str;
      continue;
    }

    // We'll have to examine the chunk byte by byte to ensure that it consists
    // of valid UTF-8 sequences, and to see if any of them might not be
    // normalized.
    //
    // Since PHP is not the fastest language on earth, some of this code is a
    // little ugly with inner loop optimizations.

    $head = '';
    $chunk = strlen($str);
    // Counting down is faster. I'm *so* sorry.
    $len = $chunk + 1;

    for ($i = -1; --$len; ) {
      $c = $str[++$i];
      if ($remaining = $tail_bytes[$c]) {
        // UTF-8 head byte!
        $sequence = $head = $c;
        do {
          // Look for the defined number of tail bytes...
          if (--$len && ($c = $str[++$i]) >= "\x80" && $c < "\xc0") {
            // Legal tail bytes are nice.
            $sequence .= $c;
          }
          else {
            if ($len == 0) {
              // Premature end of string! Drop a replacement character into
              // output to represent the invalid UTF-8 sequence.
              $result .= $unknown;
              break 2;
            }
            else {
              // Illegal tail byte; abandon the sequence.
              $result .= $unknown;
              // Back up and reprocess this byte; it may itself be a legal
              // ASCII or UTF-8 sequence head.
              --$i;
              ++$len;
              continue 2;
            }
          }
        } while (--$remaining);

        $n = ord($head);
        if ($n <= 0xdf) {
          $ord = ($n - 192) * 64 + (ord($sequence[1]) - 128);
        }
        elseif ($n <= 0xef) {
          $ord = ($n - 224) * 4096 + (ord($sequence[1]) - 128) * 64 + (ord($sequence[2]) - 128);
        }
        elseif ($n <= 0xf7) {
          $ord = ($n - 240) * 262144 + (ord($sequence[1]) - 128) * 4096 + (ord($sequence[2]) - 128) * 64 + (ord($sequence[3]) - 128);
        }
        elseif ($n <= 0xfb) {
          $ord = ($n - 248) * 16777216 + (ord($sequence[1]) - 128) * 262144 + (ord($sequence[2]) - 128) * 4096 + (ord($sequence[3]) - 128) * 64 + (ord($sequence[4]) - 128);
        }
        elseif ($n <= 0xfd) {
          $ord = ($n - 252) * 1073741824 + (ord($sequence[1]) - 128) * 16777216 + (ord($sequence[2]) - 128) * 262144 + (ord($sequence[3]) - 128) * 4096 + (ord($sequence[4]) - 128) * 64 + (ord($sequence[5]) - 128);
        } else {
          $ord = $n;
        }
        $result .= _transliteration_replace($ord, $unknown, $source_langcode);
        $head = '';
      } elseif ($c < "\x80") {
        // ASCII byte.
        $result .= $c;
        $head = '';
      } elseif ($c < "\xc0") {
        // Illegal tail bytes.
        if ($head == '') {
          $result .= $unknown;
        }
      } else {
        // Miscellaneous freaks.
        $result .= $unknown;
        $head = '';
      }
    }
  }
  return $result;
}

/**
 * Replaces a Unicode character using the transliteration database.
 *
 * @param $ord
 *   An ordinal Unicode character code.
 * @param $unknown
 *   Replacement string for characters that do not have a suitable ASCII
 *   equivalent.
 * @param $langcode
 *   Optional ISO 639 language code that denotes the language of the input and
 *   is used to apply language-specific variations.  Defaults to the current
 *   display language.
 * @return
 *   ASCII replacement character.
 */
function _transliteration_replace($ord, $unknown = '?', $langcode = NULL) {

  static $map = array();

  //GL: set language later
  /*

  if (!isset($langcode)) {
    global $language;
    $langcode = $language->language;
  }
  */

  $bank = $ord >> 8;

  if (!isset($map[$bank][$langcode])) {
    $file = './resources/transliteration-data/' . sprintf('x%02x', $bank) . '.php';  
    if (file_exists($file)) {
      $base = array();
      $variant = array();
      include $file;
      if ($langcode != 'en' && isset($variant[$langcode])) {
        // Merge in language specific mappings.
        $map[$bank][$langcode] = $variant[$langcode] + $base;
      }
      else {
        $map[$bank][$langcode] = $base;
      }
    }
    else {
      $map[$bank][$langcode] = array();
    }
  }

  $ord = $ord & 255;

  return isset($map[$bank][$langcode][$ord]) ? $map[$bank][$langcode][$ord] : $unknown;
}


1		<?php
2
3		/**
4		* @file
5		* Transliteration processing functions.
6		*/
7
8		/**
9		* Transliterates UTF-8 encoded text to US-ASCII.
10		*
11		* Based on Mediawiki's UtfNormal::quickIsNFCVerify().
12		* Based on Drupal 7 transliteration module.
13		*
14		* @param $string
15		* UTF-8 encoded text input.
16		* @param $unknown
17		* Replacement string for characters that do not have a suitable ASCII
18		* equivalent.
19		* @param $source_langcode
20		* Optional ISO 639 language code that denotes the language of the input and
21		* is used to apply language-specific variations. If the source language is
22		* not known at the time of transliteration, it is recommended to set this
23		* argument to the site default language to produce consistent results.
24		* Otherwise the current display language will be used.
25		* @return
26		* Transliterated text.
27		*/
28		function _transliteration_process($string, $unknown = '?', $source_langcode = NULL) {
		0 ignored issues – show Coding Style introduced 2016-06-30 15:05 UTC by Report Bug Copy Issue Report As per coding-style, this function should be in camelCase. CamelCase (...) is the practice of writing compound words or phrases such that each word or abbreviation begins with a capital letter. Learn more about camelCase. Loading history...
29		// ASCII is always valid NFC! If we're only ever given plain ASCII, we can
30		// avoid the overhead of initializing the decomposition tables by skipping
31		// out early.
32	8	if (!preg_match('/[\x80-\xff]/', $string)) {
33	7	return $string;
34		}
35
36	8	static $tail_bytes;
37
38	8	if (!isset($tail_bytes)) {
39		// Each UTF-8 head byte is followed by a certain number of tail bytes.
40	1	$tail_bytes = array();
41	1	for ($n = 0; $n < 256; $n++) {
42	1	if ($n < 0xc0) {
43	1	$remaining = 0;
44	1	}
45	1	elseif ($n < 0xe0) {
46	1	$remaining = 1;
47	1	}
48	1	elseif ($n < 0xf0) {
49	1	$remaining = 2;
50	1	}
51	1	elseif ($n < 0xf8) {
52	1	$remaining = 3;
53	1	}
54	1	elseif ($n < 0xfc) {
55	1	$remaining = 4;
56	1	}
57	1	elseif ($n < 0xfe) {
58	1	$remaining = 5;
59	1	}
60		else {
61	1	$remaining = 0;
62		}
63	1	$tail_bytes[chr($n)] = $remaining;
64	1	}
65	1	}
66
67		// Chop the text into pure-ASCII and non-ASCII areas; large ASCII parts can
68		// be handled much more quickly. Don't chop up Unicode areas for punctuation,
69		// though, that wastes energy.
70	8	preg_match_all('/[\x00-\x7f]+\|[\x80-\xff][\x00-\x40\x5b-\x5f\x7b-\xff]*/', $string, $matches);
71
72	8	$result = '';
73	8	foreach ($matches[0] as $str) {
74	8	if ($str[0] < "\x80") {
75		// ASCII chunk: guaranteed to be valid UTF-8 and in normal form C, so
76		// skip over it.
77	7	$result .= $str;
78	7	continue;
79		}
80
81		// We'll have to examine the chunk byte by byte to ensure that it consists
82		// of valid UTF-8 sequences, and to see if any of them might not be
83		// normalized.
84		//
85		// Since PHP is not the fastest language on earth, some of this code is a
86		// little ugly with inner loop optimizations.
87
88	8	$head = '';
89	8	$chunk = strlen($str);
90		// Counting down is faster. I'm so sorry.
91	8	$len = $chunk + 1;
92
93	8	for ($i = -1; --$len; ) {
94	8	$c = $str[++$i];
95	8	if ($remaining = $tail_bytes[$c]) {
96		// UTF-8 head byte!
97	8	$sequence = $head = $c;
98		do {
99		// Look for the defined number of tail bytes...
100	8	if (--$len && ($c = $str[++$i]) >= "\x80" && $c < "\xc0") {
101		// Legal tail bytes are nice.
102	8	$sequence .= $c;
103	8	}
104		else {
105		if ($len == 0) {
106		// Premature end of string! Drop a replacement character into
107		// output to represent the invalid UTF-8 sequence.
108		$result .= $unknown;
109		break 2;
110		}
111		else {
112		// Illegal tail byte; abandon the sequence.
113		$result .= $unknown;
114		// Back up and reprocess this byte; it may itself be a legal
115		// ASCII or UTF-8 sequence head.
116		--$i;
117		++$len;
118		continue 2;
119		}
120		}
121	8	} while (--$remaining);
122
123	8	$n = ord($head);
124	8	if ($n <= 0xdf) {
125	8	$ord = ($n - 192) * 64 + (ord($sequence[1]) - 128);
126	8	}
127		elseif ($n <= 0xef) {
128		$ord = ($n - 224) * 4096 + (ord($sequence[1]) - 128) * 64 + (ord($sequence[2]) - 128);
129		}
130		elseif ($n <= 0xf7) {
131		$ord = ($n - 240) * 262144 + (ord($sequence[1]) - 128) * 4096 + (ord($sequence[2]) - 128) * 64 + (ord($sequence[3]) - 128);
132		}
133		elseif ($n <= 0xfb) {
134		$ord = ($n - 248) * 16777216 + (ord($sequence[1]) - 128) * 262144 + (ord($sequence[2]) - 128) * 4096 + (ord($sequence[3]) - 128) * 64 + (ord($sequence[4]) - 128);
135		}
136		elseif ($n <= 0xfd) {
137		$ord = ($n - 252) * 1073741824 + (ord($sequence[1]) - 128) * 16777216 + (ord($sequence[2]) - 128) * 262144 + (ord($sequence[3]) - 128) * 4096 + (ord($sequence[4]) - 128) * 64 + (ord($sequence[5]) - 128);
138		} else {
139		$ord = $n;
140		}
141	8	$result .= _transliteration_replace($ord, $unknown, $source_langcode);
142	8	$head = '';
143	8	} elseif ($c < "\x80") {
144		// ASCII byte.
145		$result .= $c;
146		$head = '';
147		} elseif ($c < "\xc0") {
148		// Illegal tail bytes.
149		if ($head == '') {
150		$result .= $unknown;
151		}
152		} else {
153		// Miscellaneous freaks.
154		$result .= $unknown;
155		$head = '';
156		}
157	8	}
158	8	}
159	8	return $result;
160		}
161
162		/**
163		* Replaces a Unicode character using the transliteration database.
164		*
165		* @param $ord
166		* An ordinal Unicode character code.
167		* @param $unknown
168		* Replacement string for characters that do not have a suitable ASCII
169		* equivalent.
170		* @param $langcode
171		* Optional ISO 639 language code that denotes the language of the input and
172		* is used to apply language-specific variations. Defaults to the current
173		* display language.
174		* @return
175		* ASCII replacement character.
176		*/
177		function _transliteration_replace($ord, $unknown = '?', $langcode = NULL) {
		0 ignored issues – show Coding Style introduced 2016-06-30 15:05 UTC by Report Bug Copy Issue Report As per coding-style, this function should be in camelCase. CamelCase (...) is the practice of writing compound words or phrases such that each word or abbreviation begins with a capital letter. Learn more about camelCase. Loading history...
178	8	static $map = array();
179
180		//GL: set language later
181		/*
		0 ignored issues – show Unused Code Comprehensibility introduced 2016-06-30 15:05 UTC by Report Bug Copy Issue Report `61%` of this comment could be valid code. Did you maybe forget this after debugging? Sometimes obsolete code just ends up commented out instead of removed. In this case it is better to remove the code once you have checked you do not need it. The code might also have been commented out for debugging purposes. In this case it is vital that someone uncomments it again or your project may behave in very unexpected ways in production. This check looks for comments that seem to be mostly valid code and reports them. Loading history...
182		if (!isset($langcode)) {
183		global $language;
184		$langcode = $language->language;
185		}
186		*/
187
188	8	$bank = $ord >> 8;
189
190	8	if (!isset($map[$bank][$langcode])) {
191	1	$file = './resources/transliteration-data/' . sprintf('x%02x', $bank) . '.php';
192	1	if (file_exists($file)) {
193	1	$base = array();
194	1	$variant = array();
195	1	include $file;
196	1	if ($langcode != 'en' && isset($variant[$langcode])) {
197		// Merge in language specific mappings.
198		$map[$bank][$langcode] = $variant[$langcode] + $base;
199		}
200		else {
201	1	$map[$bank][$langcode] = $base;
202		}
203	1	}
204		else {
205		$map[$bank][$langcode] = array();
206		}
207	1	}
208
209	8	$ord = $ord & 255;
210
211	8	return isset($map[$bank][$langcode][$ord]) ? $map[$bank][$langcode][$ord] : $unknown;
212		}
213

seblucas / cops

Pull Request — master (#274)

transliteration.php ➔ _transliteration_replace() B

Complexity

Size

Duplication

Code Coverage

Importance

Duplication Side-by-Side

Filter issues like