Robots.txt Analyser

Analyse any website's robots.txt to understand crawling rules and scraping permissions.

Fetching and analysing robots.txt...

Quick Summary

Blocked Paths

    Allowed Paths

      Sitemaps Found

        Raw robots.txt

        
                
        

                
        
                    
        Need Help With Compliant Scraping?
        
                    
        We build scrapers that respect robots.txt and follow best practices.
        
                    Get a Free Quote →