VIPnytt / RobotsTxtParser

Completed

Push — master ( 706ec6...902e66 )

by Jan-Petter

created 2016-07-22 16:04 UTC

Changes

	2 issues (2 minor) were introduced. 1 issue was fixed.
	6 added classes/operations
		B	AllowClient::isHostListed() added
		A	CleanParamClient::detectWithCommon() added
		A	CleanParamClient::appendPath() added
		B	CleanParamClient::parse() added
		B	HostClient::getWithUriFallback() added
		A	CleanParamClient::detect() added

9.63

▶ ± 0

Head	JanPetterMG 2016-07-22 18:04 GMT+0200 Minor improvements + doc fixes - CleanParam ``detectWithCommon`` func added. Similar to ``detect``, but works even if no clean-param(s) is defined. - Speed improvements, replaced multibyte function usage with singlebyte function usage where it's safe to do so. - Removed some redundant default parameter values in some of the database handling classes. - Moved isHostListed and changed from public to private. Should never be public, as it's for internal usage only. May also be confusing to the user. - Renamed HostClient ``getWithFallback`` to ``getWithUriFallback``. The old name was a little bit too cryptic. - Increased the Cache ``clean`` delay parameter from 900 to 3600 sec.
Base	JanPetterMG 2016-07-15 17:37 UTC Various improvements: Features: - Allow Cron job with no time limit Bug-fixes: - Invalid paths sometimes override valid paths (allow/disallow/noindex directive) - URI scheme/host now always normalized to lowercase - Cron job does not return all updated entries General improvements - Increased compression ratio for the rendered robots.txt files - Some cURL options are now hardcoded to avoid potential issues - HTTP headers now parsed according to RFC 7230 section 3.2.4 - Refactored UriParser - Some database table changes, all backwards compatible. - Documentation Misc: - PHP 7.1 added for testing - Upgrade to VIPnytt\UserAgentParser v1.0