There was a fair amount of fuss made about Google’s decision to include ‘real’   numerical volume data in the keyword tool update (released 07/07/2008). Many   people shouted hoorah at a new age of openness from Google, and twice as many   huffed and puffed and dismissed it as inaccurate off hand. But almost no one   published the results of their tests with any numerical visibility.

So how do we test it? In theory we just need to find a keyword with a 100%   impression share for a search term, and see whether its impressions figure for   June matches with the figure given in the tool. But it was never going to be   that simple...   

There is no way within the Google interface (to my knowledge) to establish   if a keyword has 100% impression share, unless that keyword has been placed   is in its own adgroup. Our accounts only have one-keyword-adgroups in very rare   circumstances as, even with high volume exact matches, there are normally a   handful of terms that group nicely together, be they plurals or common variations   (such as the ever present “keyword UK”, that always gets volume but rarely justifies   its own adgroup from a quality score point of view for a uk targeted .co.uk   site).

So what keywords is it reasonable to assume have a 100% IS and match as closely   as possible to Google’s total keyword volume? In theory a keyword has to fulfil   the following requirements to be eligible for this test:   

 
• It needs   to be completely uncapped, its display unrestrained     by any budget or  adscheduling.
 
• It needs  to be active on BOTH Google and the search network, as the     keyword tool’s  figures include both.
     
• It needs  to have the same language and location settings as the tool,     which in our  case needs to be English, United Kingdom. The tool as yet     cannot be set to  include region targeting (and if the woeful inaccuracy     of the traffic  estimator tool when it comes to UK region targeting is     anything to go by,  I’m not sure I’d use it if it did!).
     
• It needs  to be running in 1^st position, at all times. This     is because the  search network includes ‘search results’ on sites that     have limited space  to display ads (eg ebay) and show only the top few     results.
     
• Finally  the keyword needs to be set to Exact Match. In theory this     isn’t an  absolute necessity, you could achieve a 100% IS on a broadmatch     keyword,  and the tool’s results are filterable by all match types, but     i think the  current ‘Expanded Broad Match’ relevancy lottery that Google     seems to be  running (perhaps a subject of future post), as well as variation     caused by   negative keywords, etc, will only add further inaccuracies     into the test.

The type of keywords that immediately spring to mind when we’re talking about   ‘always top, always displayed’ terms are those related to brand. Many of our   clients need to be shown as the top result when a search on their brand is performed,   whether due to competition against an untrademarkable brand, poor positioning   within the algorithmic listings or to promote a specific campaign. Of course,   brand terms are often low volume, but fortunately a few of our clients have   very generic keywords as their brand names, so I was able to proceed with eight   keywords of varying volume levels that should have received, as far as possible,   a 100% impression share.

I’ve hidden the identity of the keywords, but the figures are real.

                                                                                                                                                                                                                                                                                                                                                                                     

Keyword	Actual Impressions June 08	KW Tool 'Approx Search Volume: June'	difference from estimate	KW Tool 'Approx Average Search Volume'	difference from estimate
Generic Keyword A	58807	22000	167%	74000	-21%
Generic Keyword B	44397	33100	34%	74000	-40%
Generic Brand A	25088	12100	107%	27100	-7%
Generic Brand B	9704	8100	20%	8100	20%
Generic Brand C	8504	8100	5%	9100	-7%
Niche Brand A	2582	1600	61%	2400	8%
Niche Brand B	2377	1300	83%	1600	49%
Niche Brand C	1104	720	53%	1000	10%

Many of these differences are huge, with both “Generic Keyword A” and “Generic   Brand A” having more than twice the volume Google estimates. Interestingly,   the ‘average search volume’ is closer to the actual June figure with all but   one result. I wasn’t going to include the ‘average’ statistic, theorising that   it surely takes into account 12 months data to allow for full seasonality ,   but the fact it seems more accurate means it needs to be discussed (more on   this below).   

Why the inaccuracy? Well, we must take into account Google’s manipulation of   the figures. Google admits to only giving ‘approximate’ data, but goes one step   further than simply rounding - it actually groups all keywords into sets of   volumes. No matter what search query you put into the tool its volume will always   be one of a fixed set of results.

For example, with the kind of volumes we’re dealing with (between 1,000 and   100,000 impressions) Google allows only the following results from the keyword   tool:
  90500, 74000, 60500, 49500, 40500, 33100, 27100, 22200, 18100, 14800, 12100,   9900, 8100, 6600, 5400, 4400, 3600, 2900, 2400, 1900, 1600, 1300, 1000, 880,   720

Lets put those estimations against our keywords. I have rounded each actual   keyword volume DOWN (as all of the results exceed the estimations) to the closest   available result. The table now looks like this for June:   

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           

Keyword	Rounded Down June 08 Impression figure	KW Tool 'Approx Search Volume: June'	difference from estimate
Generic Keyword A	49500	22000	125%
Generic Keyword B	40500	33100	22%
Generic Brand A	22200	12100	83%
Generic Brand B	8100	8100	0%
Generic Brand C	8100	8100	0%
Niche Brand A	2400	1600	50%
Niche Brand B	1900	1300	46%
Niche Brand C	1000	720	39%

As you can see two keywords, “Generic Brand B” and “Generic Brand C”, are now   correct, but all the others were underestimated by more than one category. This   isn’t such a big issue at the lower end of the volume spectrum, where sets are   closer spaced, but for higher volume keywords this makes the data wildly inaccurate.   Generic B was only one set out, but this is still 10,000 impressions, potentially   hundreds of clicks a month.

The impression data that the tool produced for June is not accurate, even once   Google’s rounding is taken into account. Does this make it useless? Well, no,   of course not. First of all you’ll notice that, with the exception of “Generic   Keyword A”, the keyword’s relative positioning in the volume list was correct.   Secondly, the average volume was more accurate, and this is the column most   will use for all but seasonally affected campaigns.

This disparity between the accuracy of the average impressions and the June   impressions suggests to me that Google has manipulated the results. The way   in which the keywords are grouped into sets for volume might suggest they are   grouped in other ways, perhaps at a thematic level, and if June was a poor volume   month in certain sectors (which it was) Google might moderate the results in   these sectors down. Its much more dangerous for Google to overestimate volume   than underestimate volume – a new advertiser may be put off bidding on a high   volume term, or may limit their bid, budget or location targeting because of   a high volume estimate, therefore it makes sense for Google to cautiously underestimate   impression volume, as evidenced by the fact all June results were less than   the actual impressions.

The final factor is the location/language setting set by the user. How accurate   is Google’s location data? Does it identify UK volume in the same way it identifies   UK users when ads are displayed, or does it use a more simplistic method? The   only way to find out would be to look at data from accounts in other regions,   particularly the default US, and see what trends are observed. Additionally   I believe it perfectly likely that if modifications are made to the volume figures   of keyword groups, as suggested above, then these modifications are probably   based on US data and rolled out across other markets (particularly if the language   is the same).

I hope this has helped give an insight into the accuracy of the data the tool   produces. Please let me know what trends you’ve seen – i’ll update this when   the July figures are published and see if my observations carry through. I still   think this level of visibility is much better than the old relative volume bars   – if nothing else we can use the results to show a more accurate relative volume   than the old 1-5 scale. Use it, with caution, and assume more volume than estimated.