Google has threatened to block French media sites from its search results, should the French government start to charge it to refer traffic to other sites which is what is being considered.
The French have always been a bit eccentric. A colleague of mine described this search tax as being “…like the guy who invented the abacus seeking damages from IBM.”
However, should the French really want to get upset with Google, here’s something for them (and every other country in the world) to think about:
Imagine a French media site decides it wants to allow users in France to see its content but wants to block users in the US. That’s fine, there’s nothing stopping them from doing that. However, if they want to also appear in Google search results, that becomes a problem.
This is because Google generally crawls from the US. Block users with a US IP address and you’ve effectively blocked Googlebot from crawling your content.
But can’t I just make an exception for Googlebot? Maybe whitelist its IP addresses or detect its user-agent?
Nope. Google’s guidelines state that you should always show Googlebot the same as you would show a user from the same area. In this case, and indeed practically all cases, this means allowing US users to view your content.
A: Yes, however you must treat Googlebot the same as a normal user from that region. If you are blocking users in the country where Googlebot is crawling from, you must block Googlebot as well; treating Googlebot differently would be considered cloaking, a violation of our Webmaster Guidelines.
This is one reason you don’t see BBC iPlayer videos in Google’s search results. iPlayer doesn’t have a single video recognised by Google but Google shows plenty of its own YouTube videos in results for BBC show queries.
This is not only a problem for disadvantaged media companies outside the US but for Google who claim to only want to provide the best, most relevant results for users. Ignoring local sites as large and relevant as BBC iPlayer for terms such as as “top gear videos” on Google UK, leaves Google’s commitment to the best quality results open to question.
Google is still in antitrust talks with the European Commission and whilst I suspect this sort of thing isn’t part of any deal being brokered, I think it really should be. Perhaps the French should bring it up…