Noted July 4, 2019

Great to see Google has expanded its guide to robots.txt https://developers.google.com/search/reference/robots_txt

Remember robots.txt will not prevent a site from being indexed, only a noindex meta tag/HTTP header will do that. https://yoast.com/prevent-site-being-indexed/

By Peter Wilson

Peter has worked on the web for twenty years on everything from table based layouts in the 90s to enterprise grade CMS development. Peter’s a big fan of musical theatre and often encourages his industry colleagues to join him for a show or two in New York or in the West End.

Leave a comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.