3 points | by pyeri 4 days ago
1 comments
USE X-Robots-Tag: noindex to prevent files being indexed and let google determine how they crawl your site for themselves.
A nightmare scenario can result, otherwise, where you have content indexed but don't allow googlebot to crawl it. This does not end well.
https://developers.google.com/search/docs/crawling-indexing/...
USE X-Robots-Tag: noindex to prevent files being indexed and let google determine how they crawl your site for themselves.
A nightmare scenario can result, otherwise, where you have content indexed but don't allow googlebot to crawl it. This does not end well.
https://developers.google.com/search/docs/crawling-indexing/...