Noted July 4, 2019

Great to see Google has expanded its guide to robots.txt

Remember robots.txt will not prevent a site from being indexed, only a noindex meta tag/HTTP header will do that.

Published by Peter Wilson

Peter Wilson is a Senior WordPress Engineer at Human Made and contributor to WordPress core. Peter has worked on the web for twenty years on everything from table based layouts in the 90s to enterprise grade CMS development. Peter’s a big fan of musical theater and often encourages his WordPress community colleagues to join him for a show or two in New York or in the West End.

Leave a comment

Your email address will not be published. Required fields are marked *