python3-protego

Pure-Python robots.txt parser with support for modern conventions

Description

Protego parses directives set in robots.txt, including non-standard ones such as Request-Rate and Host. It also handles some typos in the directive names.

Upload more screenshots

Please help extend the collection of screenshots. Just make a screenshot and upload it here. You don't need to register or anything.

Upload a screenshot

Hint: upload an image here from your clipboard with Ctrl-V


Homepage

https://github.com/scrapy/protego


Install this software package

If the package is available for the distribution you are currently using on your computer then install the software by clicking on…

Install python3-protego