URL selection
Which URLs are not suitable for auditing?
Avoid login-only pages, admin URLs, local or intranet addresses, temporary preview links, CAPTCHA pages, image/PDF/download files, firewall-blocked pages, or pages that robots and permission rules prevent from being publicly accessed.
Key points
- The platform can analyze only publicly accessible web content. If a URL cannot be opened in an incognito browser session, the audit system usually cannot read it reliably either.
- PDFs, images, ZIP files, videos, or JSON API endpoints are not standard HTML pages, so they cannot be fully evaluated for heading structure, body content, FAQ, or page semantics.
- One-time token URLs, session URLs, and admin paths are poor SEO/GEO targets because search engines and regular users cannot access them consistently.
- If Cloudflare, WAF rules, CAPTCHA, or regional restrictions block the page, open the target page for public access or retry after the restriction is resolved.
Reference links
External sourceGoogle Search Essentialshttps://developers.google.com/search/docs/essentialsExternal sourceGoogle robots.txt referencehttps://developers.google.com/search/reference/robots_txtExternal sourceBing Webmaster Guidelineshttps://www.bing.com/webmasters/help/webmaster-guidelines-30fba23aExternal source百度搜索资源平台优化指南https://ziyuan.baidu.com/doc/index