WhitespaceTokenizer   A
last analyzed

Complexity

Total Complexity 2

Size/Duplication

Total Lines 12
Duplicated Lines 0 %

Coupling/Cohesion

Components 0
Dependencies 1

Importance

Changes 0
Metric Value
wmc 2
lcom 0
cbo 1
dl 0
loc 12
rs 10
c 0
b 0
f 0

1 Method

Rating   Name   Duplication   Size   Complexity  
A tokenize() 0 9 2
1
<?php
2
3
declare(strict_types=1);
4
5
namespace Phpml\Tokenization;
6
7
use Phpml\Exception\InvalidArgumentException;
8
9
class WhitespaceTokenizer implements Tokenizer
10
{
11
    public function tokenize(string $text): array
12
    {
13
        $substrings = preg_split('/[\pZ\pC]+/u', $text, -1, PREG_SPLIT_NO_EMPTY);
14
        if ($substrings === false) {
15
            throw new InvalidArgumentException('preg_split failed on: '.$text);
16
        }
17
18
        return $substrings;
19
    }
20
}
21