| 1 |  |  | """ | 
            
                                                                                                            
                            
            
                                    
            
            
                | 2 |  |  | Code used under MIT License from https://www.backblaze.com/b2/docs/string_encoding.html | 
            
                                                                                                            
                            
            
                                    
            
            
                | 3 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 4 |  |  | Copyright (c) 2015 Backblaze | 
            
                                                                                                            
                            
            
                                    
            
            
                | 5 |  |  | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: | 
                            
                    |  |  |  | 
                                                                                        
                                                                                     | 
            
                                                                                                            
                            
            
                                    
            
            
                | 6 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 7 |  |  | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. | 
                            
                    |  |  |  | 
                                                                                        
                                                                                     | 
            
                                                                                                            
                            
            
                                    
            
            
                | 8 |  |  | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. | 
                            
                    |  |  |  | 
                                                                                        
                                                                                     | 
            
                                                                                                            
                            
            
                                    
            
            
                | 9 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 10 |  |  | Additional code copyright George Sibble 2018 | 
            
                                                                                                            
                            
            
                                    
            
            
                | 11 |  |  | """ | 
            
                                                                                                            
                            
            
                                    
            
            
                | 12 | 1 |  | import os | 
            
                                                                                                            
                            
            
                                    
            
            
                | 13 | 1 |  | from hashlib import sha1 | 
            
                                                                                                            
                            
            
                                    
            
            
                | 14 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 15 | 1 |  | try: | 
            
                                                                                                            
                            
            
                                    
            
            
                | 16 | 1 |  |     from urllib import quote, unquote_plus | 
            
                                                                                                            
                            
            
                                    
            
            
                | 17 |  |  | except ImportError: | 
            
                                                                                                            
                            
            
                                    
            
            
                | 18 |  |  |     from urllib.parse import quote, unquote_plus | 
            
                                                                                                            
                            
            
                                    
            
            
                | 19 |  |  |  | 
            
                                                                                                            
                                                                
            
                                    
            
            
                | 20 |  |  |  | 
            
                                                                        
                            
            
                                    
            
            
                | 21 | 1 |  | def b2_url_encode(s): | 
                            
                    |  |  |  | 
                                                                                        
                                                                                            
                                                                                     | 
            
                                                                        
                            
            
                                    
            
            
                | 22 |  |  |     return quote(s.encode('utf-8')) | 
            
                                                                                                            
                            
            
                                    
            
            
                | 23 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 24 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 25 | 1 |  | def b2_url_decode(s): | 
                            
                    |  |  |  | 
                                                                                        
                                                                                            
                                                                                     | 
            
                                                                                                            
                            
            
                                    
            
            
                | 26 |  |  |     return unquote_plus(str(s)).decode('utf-8') | 
            
                                                                                                            
                            
            
                                    
            
            
                | 27 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 28 | 1 |  | def get_content_length(file): | 
                            
                    |  |  |  | 
                                                                                        
                                                                                            
                                                                                     | 
            
                                                                                                            
                            
            
                                    
            
            
                | 29 |  |  |     if hasattr(file, 'name') and os.path.isfile(file.name): | 
            
                                                                                                            
                            
            
                                    
            
            
                | 30 |  |  |         return os.path.getsize(file.name) | 
            
                                                                                                            
                            
            
                                    
            
            
                | 31 |  |  |     else: | 
            
                                                                                                            
                            
            
                                    
            
            
                | 32 |  |  |         raise Exception('Content-Length could not be automatically determined.') | 
            
                                                                                                            
                            
            
                                    
            
            
                | 33 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 34 | 1 |  | def decode_error(response): | 
                            
                    |  |  |  | 
                                                                                        
                                                                                     | 
            
                                                                                                            
                            
            
                                    
            
            
                | 35 |  |  |     try: | 
            
                                                                                                            
                            
            
                                    
            
            
                | 36 |  |  |         response_json = response.json() | 
            
                                                                                                            
                            
            
                                    
            
            
                | 37 |  |  |         return str(response.status_code) + ' - ' + str(response_json) | 
            
                                                                                                            
                            
            
                                    
            
            
                | 38 |  |  |     except ValueError: | 
            
                                                                                                            
                            
            
                                    
            
            
                | 39 |  |  |         raise ValueError(str(response.status_code) + ' - Invalid JSON Response') | 
            
                                                                                                            
                            
            
                                    
            
            
                | 40 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 41 | 1 |  | def get_part_ranges(content_length, part_size): | 
                            
                    |  |  |  | 
                                                                                        
                                                                                     | 
            
                                                                                                            
                            
            
                                    
            
            
                | 42 |  |  |     parts = [] | 
            
                                                                                                            
                            
            
                                    
            
            
                | 43 |  |  |     next_offest = 0 | 
            
                                                                                                            
                            
            
                                    
            
            
                | 44 |  |  |     while content_length > 0: | 
            
                                                                                                            
                            
            
                                    
            
            
                | 45 |  |  |         if content_length < part_size: | 
            
                                                                                                            
                            
            
                                    
            
            
                | 46 |  |  |             part_size = content_length | 
            
                                                                                                            
                            
            
                                    
            
            
                | 47 |  |  |         parts.append((next_offest, part_size)) | 
            
                                                                                                            
                            
            
                                    
            
            
                | 48 |  |  |         next_offest += part_size | 
            
                                                                                                            
                            
            
                                    
            
            
                | 49 |  |  |         content_length -= part_size | 
            
                                                                                                            
                            
            
                                    
            
            
                | 50 |  |  |     return parts | 
            
                                                                                                            
                            
            
                                    
            
            
                | 51 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 52 | 1 |  | class RangeStream: | 
                            
                    |  |  |  | 
                                                                                        
                                                                                     | 
            
                                                                                                            
                            
            
                                    
            
            
                | 53 |  |  |     """ | 
            
                                                                                                            
                            
            
                                    
            
            
                | 54 |  |  |     Wraps a file-like object (read only) and reads the selected | 
            
                                                                                                            
                            
            
                                    
            
            
                | 55 |  |  |     range of the file. | 
            
                                                                                                            
                            
            
                                    
            
            
                | 56 |  |  |     """ | 
            
                                                                                                            
                            
            
                                    
            
            
                | 57 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 58 | 1 |  |     def __init__(self, stream, offset, length): | 
            
                                                                                                            
                            
            
                                    
            
            
                | 59 |  |  |         """ | 
            
                                                                                                            
                            
            
                                    
            
            
                | 60 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 61 |  |  |         :param stream: | 
            
                                                                                                            
                            
            
                                    
            
            
                | 62 |  |  |         :param offset: | 
            
                                                                                                            
                            
            
                                    
            
            
                | 63 |  |  |         :param length: | 
            
                                                                                                            
                            
            
                                    
            
            
                | 64 |  |  |         :return: None | 
            
                                                                                                            
                            
            
                                    
            
            
                | 65 |  |  |         """ | 
            
                                                                                                            
                            
            
                                    
            
            
                | 66 |  |  |         self.stream = stream | 
            
                                                                                                            
                            
            
                                    
            
            
                | 67 |  |  |         self.offset = offset | 
            
                                                                                                            
                            
            
                                    
            
            
                | 68 |  |  |         self.remaining = length | 
            
                                                                                                            
                            
            
                                    
            
            
                | 69 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 70 | 1 |  |     def __enter__(self): | 
            
                                                                                                            
                            
            
                                    
            
            
                | 71 |  |  |         self.stream.__enter__() | 
            
                                                                                                            
                            
            
                                    
            
            
                | 72 |  |  |         return self | 
            
                                                                                                            
                            
            
                                    
            
            
                | 73 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 74 | 1 |  |     def __exit__(self, exc_type, exc_val, exc_tb): | 
            
                                                                                                            
                            
            
                                    
            
            
                | 75 |  |  |         return self.stream.__exit__(exc_type, exc_val, exc_tb) | 
            
                                                                                                            
                            
            
                                    
            
            
                | 76 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 77 | 1 |  |     def seek(self, pos): | 
                            
                    |  |  |  | 
                                                                                        
                                                                                     | 
            
                                                                                                            
                            
            
                                    
            
            
                | 78 |  |  |         self.stream.seek(self.offset + pos) | 
            
                                                                                                            
                            
            
                                    
            
            
                | 79 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 80 | 1 |  |     def read(self, size=None): | 
                            
                    |  |  |  | 
                                                                                        
                                                                                     | 
            
                                                                                                            
                            
            
                                    
            
            
                | 81 |  |  |         if size is None: | 
            
                                                                                                            
                            
            
                                    
            
            
                | 82 |  |  |             to_read = self.remaining | 
            
                                                                                                            
                            
            
                                    
            
            
                | 83 |  |  |         else: | 
            
                                                                                                            
                            
            
                                    
            
            
                | 84 |  |  |             to_read = min(size, self.remaining) | 
            
                                                                                                            
                            
            
                                    
            
            
                | 85 |  |  |         data = self.stream.read(to_read) | 
            
                                                                                                            
                            
            
                                    
            
            
                | 86 |  |  |         self.remaining -= len(data) | 
            
                                                                                                            
                            
            
                                    
            
            
                | 87 |  |  |         return data | 
            
                                                                                                            
                            
            
                                    
            
            
                | 88 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 89 | 1 |  | class StreamWithHashProgress: | 
                            
                    |  |  |  | 
                                                                                        
                                                                                     | 
            
                                                                                                            
                            
            
                                    
            
            
                | 90 |  |  |     """ | 
            
                                                                                                            
                            
            
                                    
            
            
                | 91 |  |  |     Wraps a file-like object (read-only), hashes on-the-fly, and | 
            
                                                                                                            
                            
            
                                    
            
            
                | 92 |  |  |     updates a progress_listener as data is read. | 
            
                                                                                                            
                            
            
                                    
            
            
                | 93 |  |  |     """ | 
            
                                                                                                            
                            
            
                                    
            
            
                | 94 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 95 | 1 |  |     def __init__(self, stream, progress_listener=None): | 
            
                                                                                                            
                            
            
                                    
            
            
                | 96 |  |  |         """ | 
            
                                                                                                            
                            
            
                                    
            
            
                | 97 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 98 |  |  |         :param stream: | 
            
                                                                                                            
                            
            
                                    
            
            
                | 99 |  |  |         :param progress_listener: | 
            
                                                                                                            
                            
            
                                    
            
            
                | 100 |  |  |         :return: None | 
            
                                                                                                            
                            
            
                                    
            
            
                | 101 |  |  |         """ | 
            
                                                                                                            
                            
            
                                    
            
            
                | 102 |  |  |         self.stream = stream | 
            
                                                                                                            
                            
            
                                    
            
            
                | 103 |  |  |         self.progress_listener = progress_listener | 
            
                                                                                                            
                            
            
                                    
            
            
                | 104 |  |  |         self.bytes_completed = 0 | 
            
                                                                                                            
                            
            
                                    
            
            
                | 105 |  |  |         self.digest = sha1() | 
            
                                                                                                            
                            
            
                                    
            
            
                | 106 |  |  |         self.hash = None | 
            
                                                                                                            
                            
            
                                    
            
            
                | 107 |  |  |         self.hash_read = 0 | 
            
                                                                                                            
                            
            
                                    
            
            
                | 108 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 109 | 1 |  |     def __enter__(self): | 
            
                                                                                                            
                            
            
                                    
            
            
                | 110 |  |  |         return self | 
            
                                                                                                            
                            
            
                                    
            
            
                | 111 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 112 | 1 |  |     def __exit__(self, exc_type, exc_val, exc_tb): | 
            
                                                                                                            
                            
            
                                    
            
            
                | 113 |  |  |         return self.stream.__exit__(exc_type, exc_val, exc_tb) | 
            
                                                                                                            
                            
            
                                    
            
            
                | 114 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 115 | 1 |  |     def read(self, size=None): | 
                            
                    |  |  |  | 
                                                                                        
                                                                                     | 
            
                                                                                                            
                            
            
                                    
            
            
                | 116 |  |  |         data = b'' | 
            
                                                                                                            
                            
            
                                    
            
            
                | 117 |  |  |         if self.hash is None: | 
            
                                                                                                            
                            
            
                                    
            
            
                | 118 |  |  |             # Read some bytes from stream | 
            
                                                                                                            
                            
            
                                    
            
            
                | 119 |  |  |             if size is None: | 
            
                                                                                                            
                            
            
                                    
            
            
                | 120 |  |  |                 data = self.stream.read() | 
            
                                                                                                            
                            
            
                                    
            
            
                | 121 |  |  |             else: | 
            
                                                                                                            
                            
            
                                    
            
            
                | 122 |  |  |                 data = self.stream.read(size) | 
            
                                                                                                            
                            
            
                                    
            
            
                | 123 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 124 |  |  |             # Update hash | 
            
                                                                                                            
                            
            
                                    
            
            
                | 125 |  |  |             self.digest.update(data) | 
            
                                                                                                            
                            
            
                                    
            
            
                | 126 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 127 |  |  |             # Check for end of stream | 
            
                                                                                                            
                            
            
                                    
            
            
                | 128 |  |  |             if size is None or len(data) < size: | 
            
                                                                                                            
                            
            
                                    
            
            
                | 129 |  |  |                 self.hash = self.digest.hexdigest() | 
            
                                                                                                            
                            
            
                                    
            
            
                | 130 |  |  |                 if size is not None: | 
            
                                                                                                            
                            
            
                                    
            
            
                | 131 |  |  |                     size -= len(data) | 
            
                                                                                                            
                            
            
                                    
            
            
                | 132 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 133 |  |  |             # Update progress listener | 
            
                                                                                                            
                            
            
                                    
            
            
                | 134 |  |  |             self._update(len(data)) | 
            
                                                                                                            
                            
            
                                    
            
            
                | 135 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 136 |  |  |         else: | 
            
                                                                                                            
                            
            
                                    
            
            
                | 137 |  |  |             # The end of stream was reached, return hash now | 
            
                                                                                                            
                            
            
                                    
            
            
                | 138 |  |  |             size = size or len(self.hash) | 
            
                                                                                                            
                            
            
                                    
            
            
                | 139 |  |  |             data += str.encode(self.hash[self.hash_read:self.hash_read + size]) | 
            
                                                                                                            
                            
            
                                    
            
            
                | 140 |  |  |             self.hash_read += size | 
            
                                                                                                            
                            
            
                                    
            
            
                | 141 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 142 |  |  |         return data | 
            
                                                                                                            
                            
            
                                    
            
            
                | 143 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 144 | 1 |  |     def _update(self, delta): | 
            
                                                                                                            
                            
            
                                    
            
            
                | 145 |  |  |         self.bytes_completed += delta | 
            
                                                                                                            
                            
            
                                    
            
            
                | 146 |  |  |         if self.progress_listener is not None: | 
            
                                                                                                            
                            
            
                                    
            
            
                | 147 |  |  |             self.progress_listener(self.bytes_completed) | 
            
                                                                                                            
                            
            
                                    
            
            
                | 148 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 149 | 1 |  |     def get_hash(self): | 
                            
                    |  |  |  | 
                                                                                        
                                                                                     | 
            
                                                                                                            
                            
            
                                    
            
            
                | 150 |  |  |         return self.hash | 
            
                                                                                                            
                            
            
                                    
            
            
                | 151 |  |  |  | 
            
                                                                                                            
                            
            
                                    
            
            
                | 152 | 1 |  |     def hash_size(self): | 
                            
                    |  |  |  | 
                                                                                        
                                                                                     | 
            
                                                                                                            
                                                                
            
                                    
            
            
                | 153 |  |  |         return self.digest.digest_size * 2 | 
            
                                                        
            
                                    
            
            
                | 154 |  |  |  | 
            
                        
This check looks for lines that are too long. You can specify the maximum line length.