TokenizerRegistry

Complexity

Total Complexity

1

Size/Duplication

Total Lines	8
Duplicated Lines	0 %

Importance

Changes	2
Bugs	1	Features	0

1 Method

Rating	Name	Duplication	Size	Complexity
A	__init__()	0	3	1

from six.moves import UserDict

This module should have a docstring.

The import six.moves could not be resolved.

from functools import partial

from topik.singleton_registry import _base_register_decorator


# This subclass serves to establish a new singleon instance of functions
#    for this particular step in topic modeling.  No implementation necessary.
class TokenizerRegistry(UserDict, object):

The variable object does not seem to be defined.

The variable UserDict does not seem to be defined.

    """Uses Borg design pattern.  Core idea is that there is a global registry for each step's
    possible methods
    """
    __shared_state = {}
    def __init__(self, *args, **kwargs):
        self.__dict__ = self.__shared_state

The variable self does not seem to be defined.

        super(TokenizerRegistry, self).__init__(*args, **kwargs)

The variable args does not seem to be defined.

The variable kwargs does not seem to be defined.

# a nicer, more pythonic handle to our singleton instance
registered_tokenizers = TokenizerRegistry()

The name registered_tokenizers does not conform to the constant naming conventions ((([A-Z_][A-Z0-9_]*)|(__.*__))$).

# fill in the registration function
register = partial(_base_register_decorator, registered_tokenizers)

The name register does not conform to the constant naming conventions ((([A-Z_][A-Z0-9_]*)|(__.*__))$).

The variable _base_register_decorator does not seem to be defined.

def tokenize(corpus, method="simple", **kwargs):
    """Break documents up into component words, optionally eliminating stopwords.

    Output from this function is used as input to vectorization steps.

    raw_data: iterable corpus object containing the text to be processed.
        Each iteration call should return a new document's content.
    method: string id of tokenizer to use.  For keys, see
        topik.tokenizers.registered_tokenizers (which is a dictionary of functions)
    kwargs: arbitrary dicionary of extra parameters.
    """
    return registered_tokenizers[method](corpus, **kwargs)

The variable kwargs does not seem to be defined.