Fetch and parse a site's robots.txt. Test if a specific URL is allowed or blocked for a given crawler.