Crawler Policy

Last Updated: July 2, 2026

This page describes the automated scanner used by SeoLoupe to perform SEO and technical audits. If you operate a website and have seen requests from this user agent in your logs, this page explains what it does and why.

Identification

SeoLoupeBot identifies itself with the following User-Agent string:

Mozilla/5.0 (compatible; SeoLoupeBot/1.0; +https://seoloupe.com/crawler-policy)

What triggers a scan

SeoLoupeBot is not a continuous or scheduled web crawler and does not maintain a search index. It only runs when a person manually submits a URL to be audited through SeoLoupe. Each scan is a single, on-demand event tied to that request.

What a single scan does

A scan is limited to the one URL submitted. SeoLoupeBot does not crawl deeper into a site or follow links to discover additional pages. During a scan, it may make the following requests:

• One request for the HTML of the submitted page.
• Requests for up to 28 of the scripts, stylesheets, and images directly referenced in that page's HTML, used only to estimate total page weight. These are lightweight requests (HEAD, occasionally a small ranged GET) rather than full downloads.
• Requests against up to 8 links sampled from the page, used only to estimate a broken-link rate. These links are not crawled or followed further.
• A check for the presence of /robots.txt, /sitemap.xml, /sitemap-index.xml, /llms.txt, and /ai.txt at the site root, used to report on crawler and AI-crawler readiness.
• One additional connection made directly to the submitted URL to measure DNS, TCP, and TLS connection timing. The response body is discarded as soon as the headers arrive and is not read or stored.

Rendering

SeoLoupeBot does not use a headless browser and does not execute JavaScript. All data is collected from direct HTTP responses. Content or resources that a page adds dynamically via client-side JavaScript after load are not seen, fetched, or counted.

Request volume and pacing

Each scan is a short, bounded burst of requests capped at a few seconds in total, with small per-request timeouts and limited concurrency. SeoLoupeBot does not repeatedly hit a site outside of a user-initiated scan, and does not run in the background.

Non-public addresses

SeoLoupeBot will not send requests to private, internal, loopback, or other non-public network addresses, regardless of what URL is submitted for a scan.

Blocking or excluding your domain

If you would like your domain excluded from being scanned by SeoLoupeBot, or have any other questions about this crawler, contact us at [email protected].