python3-protego
Pure-Python robots.txt parser with support for modern conventions
Description
Protego parses directives set in robots.txt, including non-standard ones such as Request-Rate and Host. It also handles some typos in the directive names.Upload more screenshots
Please help extend the collection of screenshots. Just make a screenshot and upload it here. You don't need to register or anything.
Upload a screenshotHint: upload an image here from your clipboard with Ctrl-V
Homepage
https://github.com/scrapy/protego
Install this software package
If the package is available for the distribution you are currently using on your computer then install the software by clicking on…
Install python3-protego