SEO GEO Checker

Meta Robots Tag Setup Guide | SEO GEO Checker

The meta robots tag tells search engines whether the current page can be indexed and whether links on the page can be followed. Missing it usually does not...

Problem

The meta robots tag tells search engines whether the current page can be indexed and whether links on the page can be followed. Missing it usually does not block indexing, but an explicit directive reduces ambiguity, especially for important landing pages, product pages, and content pages.

Recommended Setup

For pages that should appear in search results and pass discovery through their links, add index, follow in the <head>. This clearly communicates that the page may be indexed and its links may be crawled.

Example: <meta name="robots" content="index, follow">

Common Mistakes

- Important production pages accidentally use noindex and cannot be indexed - nofollow prevents search engines from discovering internal links - Test-environment robots directives are left in place after deployment - Meta robots and the HTTP X-Robots-Tag header contain conflicting rules

Advanced Notes

For PDFs, images, and other non-HTML assets, use the HTTP X-Robots-Tag header to control indexing. For pages that should not be indexed, such as login, signup, or admin pages, explicitly use noindex, nofollow and keep them out of your sitemap.