InMemoryOutput

Complexity

Total Complexity

17

Size/Duplication

Total Lines	54
Duplicated Lines	0 %

Importance

Changes	3
Bugs	1	Features	1

5 Methods

Rating	Name	Duplication	Size	Complexity
B	get_filtered_data()	0	8	5
B	import_from_iterable()	0	14	5
A	get_date_filtered_data()	0	4	1
A	save()	0	7	1
B	__init__()	0	13	5

from six.moves import UserDict

This module should have a docstring.

The import six.moves could not be resolved.

import types

from ._registry import register_output
from .base_output import OutputInterface


class GreedyDict(UserDict, object):

This class should have a docstring.

The variable UserDict does not seem to be defined.

The variable object does not seem to be defined.

    def __setitem__(self, key, value):
        if isinstance(value, types.GeneratorType):

The variable types does not seem to be defined.

The variable value does not seem to be defined.

            value = [val for val in value]

The variable val does not seem to be defined.

        super(GreedyDict, self).__setitem__(key, value)

The variable key does not seem to be defined.

    def __iter__(self):
        for val in self.data.values():

The Instance of GreedyDict does not seem to have a member named data.

            yield val

The variable val does not seem to be defined.

@register_output

This class should have a docstring.

This interface does not seem to be used anywhere.

The variable register_output does not seem to be defined.

class InMemoryOutput(OutputInterface):

The variable OutputInterface does not seem to be defined.

    def __init__(self, iterable=None, hash_field=None,
                 tokenized_corpora=None,
                 vectorized_corpora=None, modeled_corpora=None):
        super(InMemoryOutput, self).__init__()

        self.corpus = GreedyDict()

        if iterable:

The variable iterable does not seem to be defined.

            self.import_from_iterable(iterable, hash_field)

The variable hash_field does not seem to be defined.

        self.tokenized_corpora = tokenized_corpora if tokenized_corpora else GreedyDict()

The variable tokenized_corpora does not seem to be defined.

        self.vectorized_corpora = vectorized_corpora if vectorized_corpora else GreedyDict()

The variable vectorized_corpora does not seem to be defined.

        self.modeled_corpora = modeled_corpora if modeled_corpora else GreedyDict()

The variable modeled_corpora does not seem to be defined.

    def import_from_iterable(self, iterable, field_to_hash):
        """
        iterable: generally a list of dicts, but possibly a list of strings
            This is your data.  Your dictionary structure defines the schema
            of the elasticsearch index.
        """
        self.hash_field=field_to_hash

Exactly one space required around assignment
self.hash_field=field_to_hash
^

The variable field_to_hash does not seem to be defined.

        for item in iterable:

The variable iterable does not seem to be defined.

            if isinstance(item, basestring):

Undefined variable 'basestring'

The variable item does not seem to be defined.

The variable basestring does not seem to be defined.

                item = {field_to_hash: item}
            elif field_to_hash not in item and field_to_hash in item.values()[0]:
                item = item.values()[0]
            id = hash(item[field_to_hash])

This seems to re-define the built-in id.

The name id does not conform to the variable naming conventions ([a-z_][a-z0-9_]{2,30}$).

            self.corpus[id] = item

    # TODO: generalize for datetimes

TODO and FIXME comments should generally be avoided.

    # TODO: validate input data to ensure that it has valid year data

TODO and FIXME comments should generally be avoided.

    def get_date_filtered_data(self, field_to_get, start, end, filter_field="year"):

This method should have a docstring.

        return self.get_filtered_data(field_to_get,

The variable field_to_get does not seem to be defined.

                                      "{}<=int({}['{}'])<={}".format(start, "{}",

The variable start does not seem to be defined.

                                                                     filter_field, end))

The variable filter_field does not seem to be defined.

The variable end does not seem to be defined.

    def get_filtered_data(self, field_to_get, filter=""):

This seems to re-define the built-in filter.

        if not filter:

The variable filter does not seem to be defined.

            for doc_id, doc in self.corpus.items():

The Instance of GreedyDict does not seem to have a member named items.

                yield doc_id, doc[field_to_get]

The variable doc_id does not seem to be defined.

The variable field_to_get does not seem to be defined.

The variable doc does not seem to be defined.

        else:
            for doc_id, doc in self.corpus.items():

The Instance of GreedyDict does not seem to have a member named items.

                if eval(filter.format(doc)):

eval should generally only be used if absolutely necessary.

                    yield doc_id, doc[field_to_get]

    def save(self, filename):

Arguments number differs from overridden 'save' method

        saved_data = {"iterable": self.corpus,

The variable self does not seem to be defined.

                      "hash_field": self.hash_field,
                      "modeled_corpora": self.modeled_corpora,
                      "vectorized_corpora": self.vectorized_corpora,
                      "tokenized_corpora": self.tokenized_corpora}
        return super(InMemoryOutput, self).save(filename, saved_data)

The variable filename does not seem to be defined.

The variable saved_data does not seem to be defined.