A robots.txt file is something you create to tell search engines—typically Google—how they should crawl your website.
In simple terms, the robots.txt file tells search engines whether a page should be indexed (“index”) or not indexed (“no-index”).
It can also indicate whether they should follow or not follow links on certain pages.
How you can use robots.txt on your website
Imagine you have some hidden pages on your site that are only accessible if someone has a direct link. These are pages you probably don’t want showing up in Google search results. In that case, you would use the “no-index” directive in your robots.txt file to keep them out of the search index.
This would typically be added directly in the code on the specific hidden page.
Det er måske en smule langhåret, og medmindre man er programmør, vil man også sidde og rette i denne fil på en lidt nemmere facon.
Of course, this can get a bit technical. Unless you’re a developer, you’ll probably want a simpler way to manage these settings.
Two tools that can help you
One option is to use a plugin like Yoast SEO for WordPress. This way, you don’t need to write any code—you just tick a few boxes to tell search engines how to handle your pages.
Another helpful tool is Google’s own robots.txt tester. It’s a bit harder to find in the new version of Search Console, and that might not be a coincidence. I’ll explain why below.
Is robots.txt still relevant?
As Google becomes increasingly sophisticated, the robots.txt file isn’t as crucial as it used to be. Google is now quite good at figuring out which parts of a site to crawl and which ones to ignore. Also, directives like “no-index” are now being phased out in how Google interprets content.
You can find more about Google’s current view on robots.txt in their own documentation.
Comments