VectorizerOutput

Complexity

Total Complexity

21

Size/Duplication

Total Lines	58
Duplicated Lines	0 %

Importance

Changes	7
Bugs	4	Features	0

10 Methods

Rating	Name	Duplication	Size	Complexity
A	get_vectors()	0	3	2
A	term_frequency()	0	3	1
A	global_term_count()	0	3	1
A	__len__()	0	2	1
A	vectors()	0	3	1
D	__init__()	0	21	11
A	id_term_map()	0	3	1
A	document_term_counts()	0	3	1
A	term_id_map()	0	3	1
A	doc_lengths()	0	3	1

from collections import Counter

This module should have a docstring.

import itertools

def _accumulate_terms(tokenized_corpus):

This function should have a docstring.

    global_terms=set()

Exactly one space required around assignment
global_terms=set()
^

    document_term_counts = {}
    doc_lengths = {}
    global_term_frequency_counter = Counter()
    for doc_id, doc in tokenized_corpus:
        doc_terms = set(doc)
        global_terms.update(doc_terms)
        doc_lengths[doc_id] = len(doc)
        document_term_counts[doc_id] = len(doc_terms)
        global_term_frequency_counter.update(doc)
    id_term_map = {}
    global_term_frequency = {}
    for term_id, term in enumerate(global_terms):
        id_term_map[term_id] = term
        global_term_frequency[term_id] = global_term_frequency_counter[term]

    return id_term_map, document_term_counts, doc_lengths, global_term_frequency


class VectorizerOutput(object):

This class should have a docstring.

The variable object does not seem to be defined.

    def __init__(self, tokenized_corpus=None, vectorizer_func=None,
                 id_term_map=None, document_term_counts=None, doc_lengths=None,
                 term_frequency=None, vectors=None):
        if tokenized_corpus and vectorizer_func and not vectors:

The variable vectors does not seem to be defined.

The variable tokenized_corpus does not seem to be defined.

The variable vectorizer_func does not seem to be defined.

            iter1, iter2 = itertools.tee(tokenized_corpus)
            self._id_term_map, self._document_term_counts, self._doc_lengths, \
                self._term_frequency = _accumulate_terms(iter1)

The variable iter1 does not seem to be defined.

            self._term_id_map = {term: id

The variable id does not seem to be defined.

The variable term does not seem to be defined.

                                 for id, term in self._id_term_map.items()}
            self._vectors = vectorizer_func(iter2, self)

The variable self does not seem to be defined.

The variable iter2 does not seem to be defined.

        elif id_term_map and document_term_counts and doc_lengths and \

The variable id_term_map does not seem to be defined.

The variable doc_lengths does not seem to be defined.

The variable document_term_counts does not seem to be defined.

                term_frequency and vectors:

The variable term_frequency does not seem to be defined.

            self._id_term_map = id_term_map
            self._term_id_map = {term: id for id, term in self._id_term_map.items()}
            self._document_term_counts = document_term_counts
            self._doc_lengths = doc_lengths
            self._term_frequency = term_frequency
            self._vectors = vectors
        else:
            raise ValueError(
                "Must provide either tokenized corpora and vectorizer func, "
                "or global term collection, document term counts, and vectors.")

    def get_vectors(self):

This method should have a docstring.

        for doc_id, vector in self._vectors.items():
            yield doc_id, vector

The variable vector does not seem to be defined.

The variable doc_id does not seem to be defined.

    def __len__(self):
        return len(self._vectors)

The variable self does not seem to be defined.

    @property

The variable property does not seem to be defined.

    def id_term_map(self):

This method should have a docstring.

        return self._id_term_map

The variable self does not seem to be defined.

    @property

The variable property does not seem to be defined.

    def term_id_map(self):

This method should have a docstring.

        return self._term_id_map

The variable self does not seem to be defined.

    @property

The variable property does not seem to be defined.

    def global_term_count(self):

This method should have a docstring.

        return len(self.id_term_map)

The variable self does not seem to be defined.

    @property

The variable property does not seem to be defined.

    def document_term_counts(self):

This method should have a docstring.

        return self._document_term_counts

The variable self does not seem to be defined.

    @property

The variable property does not seem to be defined.

    def doc_lengths(self):

This method should have a docstring.

        return self._doc_lengths

The variable self does not seem to be defined.

    @property

The variable property does not seem to be defined.

    def term_frequency(self):

This method should have a docstring.

        return self._term_frequency

The variable self does not seem to be defined.

    @property

The variable property does not seem to be defined.

    def vectors(self):

This method should have a docstring.

        return self._vectors

The variable self does not seem to be defined.