About CrawlDriftBot
CrawlDriftBot is the crawler used by CrawlDrift to monitor authorized websites for technical SEO changes.
It checks server-delivered pages and HTTP signals so verified site owners can detect changes in indexability, canonicals, redirects, sitemaps, metadata, internal linking, and other technical SEO signals.
CrawlDriftBot is designed to crawl politely, respect site controls, and avoid private network access.
How to identify CrawlDriftBot
CrawlDriftBot identifies itself with the following User-Agent:
CrawlDriftBot/1.0 (+https://www.crawldrift.com/bot)Example request:
GET /example-page HTTP/1.1
Host: example.com
User-Agent: CrawlDriftBot/1.0 (+https://www.crawldrift.com/bot)CrawlDrift does not intentionally disguise the crawler as a browser, search engine bot, or another automated client.
Why CrawlDriftBot visits a website
CrawlDriftBot normally visits a domain because an authorized CrawlDrift user has:
- Added the domain to a CrawlDrift Project.
- Completed the required domain-verification process.
- Started or scheduled a crawl.
CrawlDrift is intended for monitoring public, authorized domains.
What CrawlDriftBot monitors
Depending on Project configuration and product availability, CrawlDriftBot may inspect:
- HTTP status codes and redirects
- Response headers
- Server-delivered HTML
- Titles and meta descriptions
- Canonical tags
- Robots directives
- Hreflang
- Structured data
- Heading structure
- Internal links
- Sitemap relationships
- SSL and HTTPS integrity
- GA4 and GTM tag presence
- Response timing and size metadata
CrawlDrift currently monitors server-delivered HTTP and HTML responses.
Browser rendering is not included in the current release.
Crawl politeness
CrawlDriftBot uses per-host controls to avoid placing unnecessary load on websites.
Default behavior includes:
Maximum concurrent requests per host: 2
Default request rate per host: up to 60 requests per minute
Default minimum delay between requests: 1 second
Request timeout: 10 seconds
Maximum redirects per request: 5Actual traffic may be lower depending on:
robots.txt- Server responses
Retry-Afterheaders- Project configuration
- Crawl quota
- Host cooldowns
- Platform safety controls
When a server returns 429 Too Many Requests or a temporary 503 Service Unavailable, CrawlDriftBot applies backoff and may temporarily pause requests to that host.
robots.txt
CrawlDriftBot respects robots.txt.
To block the crawler completely:
User-agent: CrawlDriftBot
Disallow: /To block a specific path:
User-agent: CrawlDriftBot
Disallow: /private-section/To allow the crawler:
User-agent: CrawlDriftBot
Disallow:Changes to robots.txt may take a short time to be reflected because robots rules can be cached.
An authorized CrawlDrift user can also pause scheduled crawling from the CrawlDrift application.
Network safety
CrawlDriftBot is not permitted to crawl private or infrastructure-only network destinations.
CrawlDrift blocks access to destinations such as:
- Loopback addresses
- Private network ranges
- Link-local addresses
- Cloud metadata services
- Internal-only hostnames
- Unsafe redirect targets
DNS and destination safety are revalidated during redirects to reduce DNS-rebinding and SSRF risks.
Redirect behavior
CrawlDriftBot may follow redirects when analyzing a page.
Every redirect remains subject to:
- Redirect limits
- Destination validation
- Private-network blocking
- Request timeout
- Per-host politeness
The crawler does not follow unlimited redirect chains.
Data handling
CrawlDrift may process the technical information required to provide monitoring, including:
- Requested URL
- HTTP status
- Response headers
- Response timing
- Redirect information
- Server-delivered HTML
- Parsed technical SEO signals
- Change evidence
Customer data is isolated by Workspace boundaries.
Credentials, authentication headers, session cookies, and crawler secrets are not intended to appear in public reports or Support diagnostics.
For more information, review the CrawlDrift Privacy Policy and Security page.
Stop or reduce CrawlDriftBot traffic
Block with robots.txt
User-agent: CrawlDriftBot
Disallow: /Pause the CrawlDrift Project
An authorized Project member can pause scheduled crawling from the CrawlDrift application.
Contact CrawlDrift
For unexpected, excessive, or unauthorized crawler activity, contact:
[email protected]Report unwanted or suspicious crawling
Email:
[email protected]Please include as much of the following information as possible:
- Affected domain
- Affected URL or path
- Approximate date and time
- Timezone
- Source IP shown in your logs
- User-Agent
- Request frequency
- Relevant server-log samples
- Whether you want crawling paused or investigated
Do not send passwords, private keys, session cookies, or authentication tokens.
CrawlDrift may temporarily pause or restrict crawling while a report is investigated.
Product Support
For questions about your CrawlDrift account, Project, crawl, or Incident, use the authenticated Support area:
https://app.crawldrift.com/supportThe abuse address is intended for crawler abuse, unauthorized activity, security-sensitive crawler reports, and network-owner requests.
[email protected]
Intended use
Use [email protected] for:
- Suspected unauthorized crawling
- Excessive request reports
- Claimed
robots.txtnon-compliance - Suspected incorrect domain verification
- Hosting provider or network-owner reports
- Reports of fake CrawlDriftBot User-Agents
- Crawler-related security reports
Do not use it for:
- General product support
- Billing questions
- Incident explanations
- Plan changes
- Feature requests
For those topics, use the authenticated CrawlDrift Support area.
Short reporting box
Report unexpected crawling
Seeing unexpected CrawlDriftBot traffic?
Email [email protected] with the affected domain, approximate timestamp, timezone, and a short server-log sample.
Do not include passwords, cookies, private keys, or authentication tokens.