ruby-webrobots

Library for creating robots.txt-aware web robots

Description

This library helps write robots.txt-compliant web robots in Ruby, based on Nokogiri's functionality.

Upload more screenshots

Please help extend the collection of screenshots. Just make a screenshot and upload it here. You don't need to register or anything.

Hint: upload an image here from your clipboard with Ctrl-V

Homepage

Install this software package

If the package is available for the distribution you are currently using on your computer then install the software by clicking on…