Completed
Push — master ( ceaeb4...21ef80 )
by Haibao
58s
created

goatools.nopen()   F

Complexity

Conditions 17

Size

Total Lines 69

Duplication

Lines 0
Ratio 0 %
Metric Value
cc 17
dl 0
loc 69
rs 2.5697

How to fix   Long Method    Complexity   

Long Method

Small methods make your code easier to understand, in particular if combined with a good name. Besides, if your method is small, finding a good name is usually much easier.

For example, if you find yourself adding comments to a method's body, this is usually a good sign to extract the commented part to a new method, and use the comment as a starting point when coming up with a good name for this new method.

Commonly applied refactorings include:

Complexity

Complex classes like goatools.nopen() often do a lot of different things. To break such a class down, we need to identify a cohesive component within that class. A common approach to find such a component is to look for fields/methods that share the same prefixes, or suffixes.

Once you have determined the fields that belong together, you can apply the Extract Class refactoring. If the component makes sense as a sub-class, Extract Subclass is also a candidate, and is often faster.

1
# Stolen from brentp:
2
# <https://github.com/brentp/toolshed/blob/master/toolshed/files.py>
3
4
import bz2
5
import gzip
6
import sys
7
import urllib
8
9
10
if sys.version_info[0] < 3:
11
    int_types = (int, long)
12
    urlopen = urllib.urlopen
13
else:
14
    int_types = (int,)
15
    basestring = str
16
    from urllib.request import urlopen
17
18
19
def nopen(f, mode="r"):
20
    r"""
21
    open a file that's gzipped or return stdin for '-'
22
    if f is a number, the result of nopen(sys.argv[f]) is returned.
23
    >>> nopen('-') == sys.stdin, nopen('-', 'w') == sys.stdout
24
    (True, True)
25
    >>> nopen(sys.argv[0])
26
    <...file...>
27
    # expands user and vars ($HOME)
28
    >>> nopen("~/.bashrc").name == nopen("$HOME/.bashrc").name
29
    True
30
    # an already open file.
31
    >>> nopen(open(sys.argv[0]))
32
    <...file...>
33
    >>> nopen(0)
34
    <...file...>
35
    Or provide nicer access to Popen.stdout
36
    >>> files = list(nopen("|ls"))
37
    >>> assert 'setup.py\n' in files or b'setup.py\n' in files, files
38
    """
39
    if isinstance(f, int_types):
40
        return nopen(sys.argv[f], mode)
41
42
    if not isinstance(f, basestring):
43
        return f
44
    if f.startswith("|"):
45
        # using shell explicitly makes things like process substitution work:
46
        # http://stackoverflow.com/questions/7407667/python-subprocess-subshells-and-redirection
47
        # use sys.stderr so we dont have to worry about checking it...
48
        p = Popen(f[1:], stdout=PIPE, stdin=PIPE,
49
                  stderr=sys.stderr if mode == "r" else PIPE,
50
                  shell=True, bufsize=-1, # use system default for buffering
51
                  preexec_fn=prefunc,
52
                  close_fds=False, executable=os.environ.get('SHELL'))
53
        if sys.version_info[0] > 2:
54
            import io
55
            p.stdout = io.TextIOWrapper(p.stdout)
56
            p.stdin = io.TextIOWrapper(p.stdin)
57
            if mode != "r":
58
                p.stderr = io.TextIOWrapper(p.stderr)
59
60
        if mode and mode[0] == "r":
61
            return process_iter(p, f[1:])
62
        return p
63
64
    if f.startswith(("http://", "https://", "ftp://")):
65
        fh = urlopen(f)
66
        if f.endswith(".gz"):
67
            return ungzipper(fh)
68
        if sys.version_info[0] < 3:
69
            return fh
70
        import io
71
        return io.TextIOWrapper(fh)
72
    f = op.expanduser(op.expandvars(f))
73
    if f.endswith((".gz", ".Z", ".z")):
74
        fh = gzip.open(f, mode)
75
        if sys.version_info[0] < 3:
76
            return fh
77
        import io
78
        return io.TextIOWrapper(fh)
79
    elif f.endswith((".bz", ".bz2", ".bzip2")):
80
        fh = bz2.BZ2File(f, mode)
81
        if sys.version_info[0] < 3:
82
            return fh
83
        import io
84
        return io.TextIOWrapper(fh)
85
86
    return {"r": sys.stdin, "w": sys.stdout}[mode[0]] if f == "-" \
87
         else open(f, mode)
88
89
90
def ungzipper(fh, blocksize=16384):
91
    """
92
    work-around to get streaming download of http://.../some.gz
93
    """
94
    import zlib
95
    uzip = zlib.decompressobj(16 + zlib.MAX_WBITS)
96
    data = uzip.decompress(fh.read(blocksize)).split("\n")
97
98
    while len(data[0]):
99
        # last chunk might not be a full line.
100
        save = data.pop()
101
        for line in data:
102
            yield line
103
        data = uzip.decompress(fh.read(blocksize)).split("\n")
104
        # first line is prepended with saved chunk from end of last set.
105
        data[0] = save + data[0]
106