Web Governance & AI Search Posture Auditor
Verify vulnerability disclosure policies (RFC 9116), protect against catastrophic Disallow: / de-indexing, and optimize your digital perimeter for modern AI search engines.
Web Governance & AI Search Posture Inspector
Probe standard root endpoints across RFC 9116 security.txt, robots.txt, sitemap.xml, and modern AI/LLM crawler rules (llms.txt / ai.txt).
Purpose of the Test
Audits public perimeter endpoints essential for ethical vulnerability disclosure (RFC 9116), search crawlability (sitemap.xml), preventing SEO disasters (robots.txt), and positioning for modern AI answer engines (llms.txt).
How It Works
Dispatches non-intrusive HTTP GET requests with TLS SNI headers to /.well-known/security.txt, /robots.txt, /sitemap.xml, and /llms.txt. Validates HTTP 200 responses and scans for destructive directives like Disallow: /.
Limitations & Specifics
Checks apex domain and common root paths over standard HTTPS. If an enterprise site enforces geo-blocking or aggressive bot-protection walls (Cloudflare Turnstile/Under Attack Mode), probes may return unverified.
How to Interpret Your Results & Next Steps for Agencies
â Security Vulnerability Disclosure (RFC 9116)
Published: Gives ethical security researchers a verified, encrypted channel to report vulnerabilities before exploitation.
Missing: Add a plain text file at /.well-known/security.txt with Contact: mailto:security@client.com and an expiration date.
â Catastrophic robots.txt Disallow Check
Warning Flag: When agencies deploy staging sites to production, they often forget to remove Disallow: / from robots.txt. This completely removes the client's site from Google Search within days.
Immediate fix: Update robots.txt immediately to allow crawler access to public content.
âšī¸ AI & LLM Search Readiness (llms.txt)
Opportunity: llms.txt is the emerging standard helping AI engines (Perplexity, ChatGPT Search, Gemini) understand a client's core services and documentation without parsing heavy client-side JavaScript.
Next step: Publish a clean Markdown summary of the client's business at /llms.txt.