Universal Search: Status Quo

In my last posting, I asked which topics you were interested in. I want to say thank you for the many replies and suggestions – some of them will need some preparation time, while others can be answered rather quickly thanks to the data available to us. One of the these, is the development or rather the status quo of the universal-search-integrations within the Google SERPs. Seeing how my last post on this subject dates back nearly a year, it is time for an update.

We will start by taking a look at the amount of SERPs with universal-search-elements, dating back to the end of 2008:


We can see how the amount has been increasing continuously and has now reached nearly 50% of all searchqueries – if, in addition, you consider how much visibility these boxes will usually get, then you will notice that there will be no way around optimizing content for the different vertical searches. Next, we have an evaluation of the different types of universal-search-integrations:


If we compare this to the beginning of last year, you will notice that the Google image search is now far more integrated. Back then, only about 10% of the universal search boxes were image related, while now, nearly 40 (!) percent of universal-search-integrations hold results from Google Images. To get a better understanding of this jump, here the progress over time of each of the different integrations, again beginning at the end of 2008:


It is clear, that the jump happened at the end of June, the beginning of Juli and that Google has kept including images at this high rate ever since. The amount of videos, maps and news shown has also gone up, while the Google shopping-search has keep at about the same level. The Google-blogsearch-integration on the other hand, has lost some ground. The overall percentages here are a little higher, since there are many keywords for which more than one integration is being shown. The following chart shows the top-10 domains that show up in the image-boxes:


Wordpress.com, which hosts the popular blogsoftware leads the way, followed by Wikimedia.org, which is the Wikipedia's multimedia-service. This diagram shows how Wordpress gained traction alongside the general jump of image-search-integrations, though since then, it has continuously kept on getting stronger:


To show this more vividly, here is a chart with some of the keywords, for which the universal-search-integrations contain pictures from Wordpress.com:


It is not hard to spot how it is not only for the lightly contested longtail (which is also in the mix) but numerous extremely traffic-strong and competitive keywords, for which the Wordpress.com-bloggers have “weaseled” their way onto the first page. So, those of you who are planning on being featured in the image-search, should try to – additionally to your own domain – post the pictures on a Wordpress.com-blog and work on making them accessible by Google. General tips on optimizing for the vertical search can be had from Martin and Andre and there will also be sessions on this topic at the upcoming Campixx.

What are you interested in?

This month has seen rather few blogposts. This was not only due to the fact that I was on vacation but also because I did not come across any topic that just desperately compelled me to write about it. I am starting to suspect that my own aspirations, to which new articles have to adhere to, might be set a little too lofty, which would mean that (maybe) some interesting topics did not get the posting they deserved.

All this is supposed to change next month and you are able to help me succeed. Just leave a short comment with the subjects that you are interested in and would like me to write about or evaluate. Please feel free to also posts “beginner-level subjects” - I often notice that those basics that I learned years ago, have changed over time and that it is quite helpful to visit them again.

Google will (probably) pull out of China

There was a huge media-echo this morning, when Google announced that they are not going to censor their Chinese searchresults anymore. A Google blogposting explained that this noticeable change in direction was triggered by an attack on Google's infrastructure: GMail-accounts belonging to human-rights activists were broken into. It is obvious that after this announcement, that Google does not have much of a future in China anymore.

Three or four years ago, I could have sworn that their motive was clear: Don't be Evil. A company like Google would not have been able to operate under these conditions without loosing their identity. Now, my first thought was: this sounds more like an excuse to pull out of China (Google with less than 20% marketshare, Baidu 70% marketshare) while saving face and getting some PR.

IndexWatch: Losers of 2009

After the Winners in the German Googleindex for 2009, we now get to the list of those who are the biggest losers of 2009, according to the SISTRIX Visibilityindex. Just as with the winning list, I will try to note trends that become apparent in the list and give a short explanation of possible causes.

Domainmoves- and Closures
Besides a few moves – like the move from premiere.de to sky.de or download.com to download.cnet.com, for example – this list contains a remarkable amount of domains that took a majority of their content offline. A few of them have already warranted their own posting in this blog over the last year: a part of Internet-history got carried to its grave when Geocities.com, one of the first and largest online freehosters, closed its doors. The Meyers.de dictionary, as well as the majority of the Brockhaus disappeared from the index. The last few weeks has seen the move from the question and answer subdomain (“iq.lycos.de”) to a new domain, which means that Lycos.de has become nearly invisible in the index since the mainpage content (“www.lycos.de”) was moved to a new domain before.

Yellow Pages
The large amount of yellow-pages, phone-directories and similar projects on this list is surely another noticeable trend. Besides such rather obvious Google Penalties like those for flix.de or lokal.ws for example, I am intrigued by the development of dialo.de: founded in 2006, purchased already by a phone-book-publishing-house in 2007, the site has continuously lost ground. It seems that they have completely sunk to insignificance in 2009. Not only is their presence within Google nearly impossible to measure, even traffic-indicators like Alexa and Google Trends are unable to detect any life on the domain. It is unknown if they have prematurely capitulated to the Google Yellow-Page-integration via Universal Search or if they were unable to conceive a way to get users to visit the page. I would not be surprised if the remaining sites in this sector will have a hard year ahead of them – Google does not only want to be a searchengine, but wants to neutralize as many stopovers between searchquery and searchresult. And phone- as well as yellow-pages belong to these stopover, too.

It is rather uncommon for a domain to show up in this list two years in a row: vanityfair.de managed to do this though. After losing many interesting rankings in 2008, they managed to make another noticeable step backwards around mid-2009 – maybe it would be better to close down the page altogether. Kijiji had to transfer a large chunk of their content on to the kleinanzeigen.ebay.de subdomain – a decision that has not really paid off from a SEO-point-of-view, but let's see how this will unfold. CMA.de (“Bestes vom Bauern”) was successful in having nearly all of their content disappear from the index: after the domain could be found in the top-10 for a number of food-related keywords (“lammfleisch”, “rezepte”, “gulasch”) at the end of last year, it seems that a relaunch made the site only show up for queries for “CMA” - but this should not be a problem anymore after the decision to close its doors.

IndexWatch: Winners of 2009

2009 has passed – time to take a look at how the Googleindex has changed in 2009. Just like last year, I will show both the winners and losers for 2009, based on the SISTRIX Visibilityindex. I will also highlight and comment on interesting developments – especially if they involve more than one domain. This blogpost will start with the Googleindex winners of 2009:

Quality Pays Off
While last year, we had quite a few scrapers and rather unconventional “content-aggregators” on the winning side, there has been a noticeable decline in the number of such domains in this list for 2009. While there may still be some exceptions here, like the social-bookmarking-site publishr.de or cylex-telefonbuch.de, which have a rather questionable added value for the displayed data-quality. Though all things considered, this year's list is better than last year's. We can only speculate on how much user-behavior played any role in this but the consensus is that quality pages are by far easier to optimize.

Optimization for Brandnames
Even though the competition for generic terms like “credit” or “cell phone” has been plenty and will probably only get more heated since the Brand-update shortly before Christmas, this usually looks a lot different for the brandnames with the most traffic. A multitude of sites on the winning side seem to have used just that to their advantage this year by targeting those visitors that type in brandnames into Google. The actual brand will probably take the top spot and most of the traffic but the rest of the visitors will usually have rather high conversion-rates which also makes them popular. tageszeitung-24.de is a nice example for this: the domain ranks within the top-10 for a number of newspapers (“dresdner morgenpost”, “coburger tageblatt” or the “neumarkter tagblatt”) and also offers appropriate newspaper-subscriptions.

Existing Content Gets Optimized
Seeing how the large publishing houses have, by now, made most of the content from their online-presences available for Google – which are also ranking quite well – 2009 also saw a number of smaller publishing houses and content-producers follow suit. Two domains from this list are exemplary, schoener-wohnen.de and gabler.de: the first domain is already ranking for terms like bathroom (1), living (5) and tables (8), while the second domain – a publishing house for professional literature – can even be found in front of Wikipedia with their economic-lexicon for keywords like “gains tax” (2) or “double accounting” (1).

SEO is Expensive
The days in which a bright guy could take a whole e-commerce site and push it to the top by himself and with little costs are over: today, the contested terms often need a SEO-team that works on tasks like onpage-optimization, linkpurchase creation of added value, which people link to freely and other SEO-tasks to get ahead. Monthly budgets in the five- or six-figure range are no exceptions anymore – money that has to be generated again, which is shown nicely by the number of e-commerce-domains that are on the winning side.

PageRank-Update 12/2009

Google is updating the PageRank-values on their datacenters, just as 2009 is drawing to a close. The roll-out is going on at the moment and some datacenters like 209.85.129.99 or 74.125.1.99 are already showing the new values. Those of you who do not want to wait for the toolbar to start displaying these new values, can use our PageRank-Query instead. At the moment, I am running an evaluation of exceptional winners or losers and will update this post accordingly.

Merry Christmas

Holding with tradition, my Christmas greetings come rather late but still from the bottom of my heart. I wish a merry Christmas and a few restful days to all of our readers. I am looking forward to next year, where I believe that we will see a lot of interesting innovations and developments in many areas pertinent to onlinemarketing.

Visibilityindex for everyone

In the two years since its introduction, the SISTRIX Visibilityindex turned into one of the most important indicators for how a domain is performing in the Googleindex. A while ago, I made a post about how this Index is being calculated each week – since then, there have been some interesting additions: Hitmeister for example, shows the correlation between the Visibilityindex and Googletraffic, Hanns has some thoughts on the best amount of pages a domain should have in the index and SEO-Scene gives an insight into how much one link can change the visibility within Google.

Up until now, this value was only accessible to SISTRIX Toolbox customers but, as sort of an early Christmas present, I want to make this tool available to everyone for free. Just visit this page, enter the domain into the form, hit enter and the Visibilityindex for the domain will be shown.

For those who would like to have the Visibilityindex on hand while they surf the web, we have developed the SISTRIX Toolbar. This Firefox plug-in sits in the status-bar of the browser and shows the SISTRIX Visibilityindex as well as other, interesting indicators for the current domain or URL. You can download the free toolbar here.

The Brands are comming

This monday has (finally?) seen some movement within the Google-SERPs, which go well above the usual EverFlux: domains for known brands are going up the ranks for a number of strong keywords. To make this more accessible, here are a few examples. The keywords are up front, followed by the brandname with the change in rankings for the previous week:

Keyword	New Branddomains
jobs	arbeitsagentur.de (New)
computer	alternate.de (New), acer.de (+10)
handys	t-mobile.de (New), vodafone.de (+30), nokia.de (New), sonyericsson.com (+50)
baby	pampers.de (+34)
drucker	canon.de (New), brother.de (New), lexmark.de (+10)
dsl	alice-dsl.de (+10), 1und1.de (+16)

And there are even more examples to be found. The common denominator for all of these sites is that those sites, that are rising through the ranks, are not optimized by SEO-criteria: oftentimes, they do not even have the keyword in the title-tag as well as other characteristics that would speak against a “natural” ranking.

A very similar change has already caused a stir in the USA a few month ago. They also had well-known (offline)-brands rise through the ranks into the top-10 for no discernible reason. It seems, that close to Christmas, this change has also found it's way to Germany. Merry Christmas with #pipxmasupdate.

Confusion about First Click Free

Ever since both the Hamburger Abendblatt as well as the Berliner Morgenpost started to function as guinea-pigs for paid content for the publishing-houses online presence at the beginning of the week, my feedreader is bristling with opinions and evaluations on this issue. I do not want to say anything about this contentwise, but I do want to write something on the technical implementation, seeing how there are obviously still some uncertainties here.

As far as publishing-houses go, Google is accommodating them more than any other content-distributor on the Internet. Danny Sullivan (from Searchengineland) has summarized this nicely a while ago. Part of this accommodation goes by the name of “First Click Free”: when a user is send to a newspapers site by Google, then they can read the first article for free and have to pay for those thereafter. Usually this is accomplished that users, which come from Google, are identified through their referrer and will then have to pay from the second pageview on.

Evil Cloaking?
For all of the pages to be entered into Google's websearch as well as Google News, the Googlebot needs to be able to crawl all of those pages without restrictions. For this, the searchenginecrawler needs to get a different version of the page than the human visitor: this is called cloaking. In the past, cloaking was a rather widespread phenomenon in the SEO-scene, but for the past few years, the advantages are so slim or nonexistent, that most will just abandon the practice. Now, when netzpolitik.org pushes the cloaking of both sides into the gray corner in a posting (“... I would have though that 'cloaking' would still lead to a sites exclusion from the searchengine result pages (SERPs).”), they are not thinking far enough ahead: in this case, there is no cloaking for the purpose of gaining some advantage in the SERPs but they are using a feature for a potential mentearization of the site with the expressed permission from Google.

Clumsy Cloaking
Another reason for postings (for example from Chip.de or Carta.info) takes into account the technical realization of the Bot/Human recognition. At the moment, the Googlebots footprint in the logfiles looks like this:

66.249.71.13 - - [16/Dec/2009:13:05:13 +0100] "GET /news/ HTTP/1.1" 200 16199 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Up front the IP-Address, then some stuff that is of no importance to us and at the end, the User-Agent. There is the possibility to implement cloaking based on the IP-Address, the User-Agents or a combination of both. At least the Abendblatt seems to have decided to just consider the User-Agent. This is somewhat clumsy because this value can be changed by users as they wish (for example with a Firefox plugin), which then lets the visitor browse through all of the content on the site. It would be better to use a combination between IP-Address and User-Agent: the big searchengines are, for years now, offering a well established procedure for this. For every access where the User-Agent shows a visit by a searchengine, the first step checks the reverse-DNS-entry for the IP-Address:

beus@helios:~$ host 66.249.71.13
Name: crawl-66-249-71-13.googlebot.com

If a hostname with the googlebot.com domain resolves, it is resolved back to the IP-Address:

beus@helios:~$ host crawl-66-249-71-13.googlebot.com
crawl-66-249-71-13.googlebot.com A 66.249.71.13

In this case, everything looks fine: the same IP-Address as in the beginning resolves in the end. This can be used to keep users from getting to pages that are only for the searchengine, just because they changed their User-Agent. Incidentally, this subject matter is not that much of a new thing …