Completed
Pull Request — master (#145)
by Freek
01:32
created

PHP Classes Changes

No PHP Classes changes.

PHP Operations Changes

A Crawler::mayIndex() added
A Crawler::mayFollow() added
A Crawler::ignoreRobots() added
A Crawler::respectRobots() added
A Crawler::createRobotsTxt() added