More than 250 websites now block Google-Extended, the “standalone product token” Google introduced Sept. 28 to let you block Bard, Vertex AI generative APIs and future generations of ****** from accessing your content. That’s according to research shared exclusively with Search Engine Land by the Detailed.com team. Why we care. There has been much debate and discussion … Read more
X, well, Twitter.com, is now blocking Bing Search, specifically Bingbot, from crawling and accessing content posted on Twitter.com, on the X platform. Twitter specifically added to its robots.txt file a directive to disallow Bingbot from crawling the content on its platform. Here is a screenshot of its robots.txt file: This was spotted first, I believe, … Read more
At least 26 of the top 100 most popular websites – and 242 of the top 1,000 – are now blocking GPTBot, the web crawler OpenAI introduced Aug. 7, according to an updated analysis. Why we care. To block or not to block ChatGPT? That has been a big question for many SEOs because ChatGPT does … Read more
January 27, 2015 | Posted by Shiri Berzack In the past, Google has warned against blocking Googlebot from crawling specific aspects of your website, and now the warning has been extended to mobile sites. Google’s Zineb Ait Bahajji announced on Google Plus, that blocking Googlebot for Smartphones will have a negative effect on … Read more
Google’s John Mueller answered a question in Reddit about whether Google indexes content placed before the main content as part of the post. John’s answer addressed part of the question but left the overall question unanswered. Nevertheless, there is a solution for the person’s question. Does GoogleBot “Collect” Content Before the Post? The person asking … Read more
I blocked two of our ranking pages using robots.txt. We lost a position here or there and all of the featured snippets for the pages. I expected a lot more impact, but the world didn’t end. Warning I don’t recommend doing this, and it’s entirely possible that your results may be different from ours. I was … Read more
421
229
714
826
543