Tokenizer::tokens()   F
last analyzed

Complexity

Conditions 31
Paths 679

Size

Total Lines 126
Code Lines 93

Duplication

Lines 0
Ratio 0 %

Code Coverage

Tests 84
CRAP Score 31.0905

Importance

Changes 4
Bugs 3 Features 0
Metric Value
cc 31
eloc 93
c 4
b 3
f 0
nc 679
nop 0
dl 0
loc 126
ccs 84
cts 88
cp 0.9545
crap 31.0905
rs 0.4458

How to fix   Long Method    Complexity   

Long Method

Small methods make your code easier to understand, in particular if combined with a good name. Besides, if your method is small, finding a good name is usually much easier.

For example, if you find yourself adding comments to a method's body, this is usually a good sign to extract the commented part to a new method, and use the comment as a starting point when coming up with a good name for this new method.

Commonly applied refactorings include:

1
<?php
2
declare(strict_types=1);
3
4
namespace JsonDecodeStream;
5
6
use Generator;
7
use IteratorAggregate;
8
use JsonDecodeStream\Exception\TokenizerException;
9
use JsonDecodeStream\Internal\SourceBuffer;
10
11
class Tokenizer implements IteratorAggregate
12
{
13
    protected const WHITESPACE_CHARS = " \t\r\n";
14
15
    protected const LITERALS = [
16
        Token::TRUE  => true,
17
        Token::FALSE => false,
18
        Token::NULL  => null,
19
    ];
20
21
    /** @var SourceBuffer|string[] */
22
    protected $buffer;
23
24
    /** @var int|null */
25
    protected $lineNumber;
26
    /** @var int|null */
27
    protected $charNumber;
28
29 107
    public function __construct(SourceBuffer $buffer)
30
    {
31 107
        $this->buffer = $buffer;
32 107
    }
33
34
    /**
35
     * @return \Traversable|Token[]
36
     * @psalm-return \Traversable<Token>
37
     * @throws TokenizerException
38
     */
39 52
    public function getIterator()
40
    {
41 52
        return $this->tokens();
42
    }
43
44
    /**
45
     * @return Generator|Token[]
46
     * @psalm-return \Traversable<Token>
47
     * @throws TokenizerException
48
     */
49 107
    public function tokens()
50
    {
51
        /** @var string|null $number collecting chars of numeric token */
52 107
        $number = null;
53
        /** @var string|null $string collecting chars of string token */
54 107
        $string = null;
55
        /** @var int $stringSlashes how many sequentative slashes are found in string */
56 107
        $stringSlashes = 0;
57
        /** @var string|null $whitespace collecting chars of whitespace token */
58 107
        $whitespace = null;
59
        /** @var string|null $literalAwaited holding the string for awaited literal */
60 107
        $literalAwaited = null;
61
        /** @var string|null $literal collecting chars of literal token */
62 107
        $literal = null;
63
64 107
        $this->lineNumber = 1;
65 107
        $this->charNumber = 1;
66 107
        foreach ($this->buffer as $char) {
67 107
            if ($char == "\n") {
68 63
                $this->lineNumber++;
69 63
                $this->charNumber = 1;
70
            } else {
71 106
                $this->charNumber++;
72
            }
73
74 107
            if ($number !== null) {
75 72
                if (strpos('0123456789+-Ee\.', $char) !== false) {
76 41
                    $number .= $char;
77 41
                    continue;
78
                }
79 52
                $parsedNumber = $this->parseNumber($number);
0 ignored issues
show
Bug introduced by
$number of type null is incompatible with the type string expected by parameter $number of JsonDecodeStream\Tokenizer::parseNumber(). ( Ignorable by Annotation )

If this is a false-positive, you can also ignore this issue in your code via the ignore-type  annotation

79
                $parsedNumber = $this->parseNumber(/** @scrutinizer ignore-type */ $number);
Loading history...
80 52
                $number = null;
81 52
                yield $this->createToken(Token::NUMBER, $parsedNumber);
82
            }
83
84 107
            if ($string !== null) {
85 62
                if ($char == '\\') {
86 7
                    $string .= $char;
87 7
                    $stringSlashes++;
88 7
                    continue;
89 62
                } elseif ($char == '"' && ($stringSlashes % 2) == 0) {
90 62
                    $parsedString = $this->parseString($string);
0 ignored issues
show
Bug introduced by
$string of type null is incompatible with the type string expected by parameter $string of JsonDecodeStream\Tokenizer::parseString(). ( Ignorable by Annotation )

If this is a false-positive, you can also ignore this issue in your code via the ignore-type  annotation

90
                    $parsedString = $this->parseString(/** @scrutinizer ignore-type */ $string);
Loading history...
91 62
                    yield $this->createToken(Token::STRING, $parsedString);
92 62
                    $string = null;
93 62
                    $stringSlashes = 0;
94 62
                    continue;
95
                } else {
96 61
                    $string .= $char;
97 61
                    $stringSlashes = 0;
98 61
                    continue;
99
                }
100
            }
101
102 107
            if ($whitespace !== null) {
103 69
                if (strpos(static::WHITESPACE_CHARS, $char)) {
104 1
                    $whitespace .= $char;
105 1
                    continue;
106
                } else {
107 68
                    yield $this->createToken(Token::WHITESPACE, $whitespace);
108 68
                    $whitespace = null;
109
                }
110
            }
111
112 107
            if ($literalAwaited !== null) {
113 26
                if (strpos($literalAwaited, $char) !== false) {
0 ignored issues
show
Bug introduced by
$literalAwaited of type null is incompatible with the type string expected by parameter $haystack of strpos(). ( Ignorable by Annotation )

If this is a false-positive, you can also ignore this issue in your code via the ignore-type  annotation

113
                if (strpos(/** @scrutinizer ignore-type */ $literalAwaited, $char) !== false) {
Loading history...
114 26
                    $literal .= $char;
115
                } else {
116
                    throw TokenizerException::unexpectedToken($literal, $this->lineNumber, $this->charNumber);
0 ignored issues
show
Bug introduced by
$literal of type null is incompatible with the type string expected by parameter $token of JsonDecodeStream\Excepti...tion::unexpectedToken(). ( Ignorable by Annotation )

If this is a false-positive, you can also ignore this issue in your code via the ignore-type  annotation

116
                    throw TokenizerException::unexpectedToken(/** @scrutinizer ignore-type */ $literal, $this->lineNumber, $this->charNumber);
Loading history...
117
                }
118
119 26
                if (strlen($literal) == strlen($literalAwaited)) {
0 ignored issues
show
Bug introduced by
$literalAwaited of type null is incompatible with the type string expected by parameter $string of strlen(). ( Ignorable by Annotation )

If this is a false-positive, you can also ignore this issue in your code via the ignore-type  annotation

119
                if (strlen($literal) == strlen(/** @scrutinizer ignore-type */ $literalAwaited)) {
Loading history...
120 26
                    if ($literal === $literalAwaited) {
121 26
                        yield $this->createToken($literalAwaited, static::LITERALS[$literalAwaited]);
0 ignored issues
show
Bug introduced by
$literalAwaited of type void is incompatible with the type string expected by parameter $tokenId of JsonDecodeStream\Tokenizer::createToken(). ( Ignorable by Annotation )

If this is a false-positive, you can also ignore this issue in your code via the ignore-type  annotation

121
                        yield $this->createToken(/** @scrutinizer ignore-type */ $literalAwaited, static::LITERALS[$literalAwaited]);
Loading history...
122 26
                        $literalAwaited = null;
123 26
                        $literal = null;
124
                    } else {
125
                        throw TokenizerException::unexpectedToken($literal, $this->lineNumber, $this->charNumber);
126
                    }
127
                }
128 26
                continue;
129
            }
130
131 107
            if (strpos(static::WHITESPACE_CHARS, $char) !== false) {
132 73
                $whitespace = $char;
133 102
            } elseif ($char == "{") {
134 56
                yield $this->createToken(Token::OBJECT_START);
135 102
            } elseif ($char == "}") {
136 56
                yield $this->createToken(Token::OBJECT_END);
137 100
            } elseif ($char == "[") {
138 51
                yield $this->createToken(Token::ARRAY_START);
139 100
            } elseif ($char == "]") {
140 51
                yield $this->createToken(Token::ARRAY_END);
141 98
            } elseif ($char == ":") {
142 55
                yield $this->createToken(Token::KEY_DELIMITER);
143 97
            } elseif ($char == ",") {
144 63
                yield $this->createToken(Token::COMA);
145 96
            } elseif ($char == '"') {
146 62
                $string = '';
147 79
            } elseif (strpos('+-0123456789', $char) !== false) {
148 74
                $number = $char;
149 26
            } elseif ($char == 't') {
150 20
                $literalAwaited = Token::TRUE;
151 20
                $literal = $char;
152 8
            } elseif ($char == 'f') {
153 7
                $literalAwaited = Token::FALSE;
154 7
                $literal = $char;
155 7
            } elseif ($char == 'n') {
156 7
                $literalAwaited = Token::NULL;
157 7
                $literal = $char;
158
            } else {
159
                throw TokenizerException::unexpectedCharacter($char, $this->lineNumber, $this->charNumber);
160
            }
161
        }
162
163 107
        if ($string !== null || $literal !== null) {
164
            throw TokenizerException::malformedString($string ?? $literal, $this->lineNumber, $this->charNumber);
0 ignored issues
show
Bug introduced by
It seems like $string ?? $literal can also be of type null; however, parameter $string of JsonDecodeStream\Excepti...tion::malformedString() does only seem to accept string, maybe add an additional type check? ( Ignorable by Annotation )

If this is a false-positive, you can also ignore this issue in your code via the ignore-type  annotation

164
            throw TokenizerException::malformedString(/** @scrutinizer ignore-type */ $string ?? $literal, $this->lineNumber, $this->charNumber);
Loading history...
165
        }
166
167 107
        if ($number !== null) {
168 22
            $parsedNumber = $this->parseNumber($number);
169 16
            yield $this->createToken(Token::NUMBER, $parsedNumber);
170 16
            $number = null;
0 ignored issues
show
Unused Code introduced by
The assignment to $number is dead and can be removed.
Loading history...
171
        }
172
173 101
        if ($whitespace !== null) {
174 17
            yield $this->createToken(Token::WHITESPACE, $whitespace);
175
        }
176 101
    }
177
178
    /**
179
     * @param string $number
180
     * @return float|int|null
181
     * @throws TokenizerException
182
     */
183 74
    protected function parseNumber(string $number)
184
    {
185 74
        if (preg_match_all('/^[+-]?(0|[1-9][0-9]*)$/', $number)) {
186 56
            return intval($number);
187 22
        } elseif (preg_match('/^[+-]?(0|[1-9][0-9]*)(\.[0-9]+)?([eE][-+]?([0-9]+))?$/', $number)) {
188 16
            return doubleval($number);
189
        } else {
190 6
            throw TokenizerException::malformedNumber($number, $this->lineNumber, $this->charNumber);
191
        }
192
    }
193
194
    /**
195
     * @param string $string
196
     * @return string|null
197
     * @throws TokenizerException
198
     */
199 62
    protected function parseString(string $string)
200
    {
201 62
        $parsed = json_decode('"' . $string . '"');
202 62
        if (!is_string($parsed)) {
203
            throw TokenizerException::malformedString($string, $this->lineNumber, $this->charNumber);
204
        }
205 62
        return $parsed;
206
    }
207
208
    /**
209
     * @param string $tokenId
210
     * @param mixed  $value
211
     * @return Token
212
     */
213 101
    protected function createToken(string $tokenId, $value = null): Token
214
    {
215 101
        return new Token($tokenId, $value, $this->lineNumber, $this->charNumber);
0 ignored issues
show
Bug introduced by
It seems like $this->lineNumber can also be of type null; however, parameter $lineNumber of JsonDecodeStream\Token::__construct() does only seem to accept integer, maybe add an additional type check? ( Ignorable by Annotation )

If this is a false-positive, you can also ignore this issue in your code via the ignore-type  annotation

215
        return new Token($tokenId, $value, /** @scrutinizer ignore-type */ $this->lineNumber, $this->charNumber);
Loading history...
Bug introduced by
It seems like $this->charNumber can also be of type null; however, parameter $charNumber of JsonDecodeStream\Token::__construct() does only seem to accept integer, maybe add an additional type check? ( Ignorable by Annotation )

If this is a false-positive, you can also ignore this issue in your code via the ignore-type  annotation

215
        return new Token($tokenId, $value, $this->lineNumber, /** @scrutinizer ignore-type */ $this->charNumber);
Loading history...
216
    }
217
}
218