# p0dalirius/robotstester

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/p0dalirius-robotstester).**

48 stars · 4 forks · Python · GPL-3.0

## Links

- GitHub: https://github.com/p0dalirius/robotstester
- Homepage: https://podalirius.net/
- awesome-repositories: https://awesome-repositories.com/repository/p0dalirius-robotstester.md

## Description

This Python script can enumerate all URLs present in robots.txt files, and test whether they can be accessed or not.

## Tags

### Part of an Awesome List

- [Reconnaissance and Dorking](https://awesome-repositories.com/f/awesome-lists/security/reconnaissance-and-dorking.md) — Enumerate and test accessibility of URLs found in robots files.