neustar.biz

Super Bowl Advertisers: Rating their Website Performance

Written by Alan Dyke and Diem Shin

The Baltimore Ravens won a truly memorable Super Bowl. But who were the winners and losers among companies investing millions on TV ads to drive website traffic and brand awareness? We’ll let others pass judgment on all that high-priced creative and share with you instead some juicy website performance nuggets. Neustar set up website monitors on over 25 of the top Super Bowl advertisers’ sites, gauging website performance every 10 minutes from Washington, DC and San Francisco.

Those sites were:

  • Anheuser Busch
  • Audi
  • Best Buy
  • Bud Light
  • Budweiser
  • Cars.com
  • Century 21
  • Chrysler
  • CokeChase
  • CocaCola
  • Ford
  • FritoLays
  • Gildan’s
  • GoDaddy
  • Hyundai
  • Kia Motors
  • M&M
  • Mercedes
  • Mondelez International
  • NFL.com
  • Pepsi
  • Samsung
  • Sketchers
  • SodaStream
  • TacoBell
  • Toyota
  • Volkswagen
  • GetCrackin.com                              (Wonderful Pistachios)
 

About the Website Monitors:

All our monitors were set up using one megabyte per second bandwidth throttle, and with 70ms latency, representative of a typical home user. Timeouts of 30 seconds were set up on each website.  For Toyota, Kia, and Hyundai, timeout restriction was relaxed because these sites were failing within the 30 second time limit.  We also set up a three strikes policy, so that any time a monitor detected a website error it would immediately validate the error with two more tests before declaring a failure.

Studies have shown that users will abandon websites if they don’t load within 3 seconds or less.  There is substantial research showing load time differences as small as .1 second result in lost sales and users,  indicating that website performance should be a priority.

The Results: Coke’s Problematic Coke Chase Website

Similar to last year, most sites performed well under stress.  However, there are some noteworthy mentions.   Coke, this year, wanted to involve the web audience in selecting the ending of the commercial.  Our Pre-Game research did not find a www.cokechase.com; however, we were able to start a website monitor just a few minutes after the commercial aired.  Neustar was not able to pick up a successful sample with their website.  The reason is that the website constantly timed out after 30 seconds. However that’s only part of the story.  The slowness was primarily due to a single object http://www.cokechase.com/config.json.

coke Super Bowl Advertisers: Rating their Website Performance

Although our monitors pronounced the web site dead after 30 seconds, our 12Mb/s web connection took 1.6 minutes to not load causing the page render to stall until after the object timed out. The object had just 25 bytes of content (and 327 bytes of headers ) and said “mode:in-game”.  Participants wanting to vote for their choice of Cowboys, Showgirls or Badlanders, were staring almost endlessly at the following:

coke2 Super Bowl Advertisers: Rating their Website Performance

About half of the monitors during the second half received an HTTP 504 error (resource unavailable). As if that wasn’t bad enough for Coke, their home website http://www.cokeacola.com also went down for hours.  This was probably due to being on the same infrastructure as the cokechase.com website.  While there was no revenue lost from the site going down, there is brand damage.  Two years in a row of their site failing during the Super Bowl cannot be good.

coke3 Super Bowl Advertisers: Rating their Website Performance

And to rub salt in Cokes’ wounds, Pepsi sponsored a fabulous half time show and kept their website up for the whole event.

Pepsi Super Bowl Advertisers: Rating their Website Performance

The Results: Gildan Website Outage

Another failure, albeit less spectacular than Coke’s was clothing retailer Gildan Activewear.  Gildan’s Edgy commercial aired at 18:25 PST and was immediately followed by a short outage on their website.

Gildan Super Bowl Advertisers: Rating their Website Performance

The Results: Taco Bell’s New (Slow) Website

Taco Bell also released a new version of their website at 3pm PST, just before the start of the Super Bowl, resulting in a larger, and slower loading website.  The newer version increased load time from 25 seconds to over 30 seconds and often timed out before completion.

Websites Are Getting Bigger, More Difficult to Load

Setting aside the small number of site failures, there is an upward trend in size of websites. Last year, the average Super Bowl site monitored was 1.36 MBytes, and required 66 requests to load.

This year, we recorded an average of 1.53Mb(up 12.5 percent), and 81 web requests (up 23 percent). Strikingly, of those companies monitored for 2012 and 2013 Super Bowls, their average websites were marginally smaller this year.   Leading the increase was M&M who transformed their slim line 220Kbyte website of 2012 into a substantial 1.44Mb (increase of 350%).

This year we added a new feature showing YSlow scores for every sample we take.  The trend from 2012 to 2013 is a disappointing one for the online user.   For example, Audi in Super Bowl 2012 had an outstanding A rating on their website.  This year they dropped to a D rating. The overall trend can be seen in the charts below:

YSlow scores for monitored websites.

yslow Super Bowl Advertisers: Rating their Website Performance Below is a comparison chart for sites monitored for the 2012 and 2013 Super Bowls:

comparison Super Bowl Advertisers: Rating their Website Performance While the super bowl marketing teams are celebrating another successful Super Bowl, spare a thought for the web engineers and operators who kept their web sites available and responsive on one of the year’s biggest traffic surges.

Lessons learned…test, test and test before a big event!!!

dot Super Bowl Advertisers: Rating their Website Performance
Alan Dyke

About Alan Dyke

Alan Dyke has a Ph.D. from the University of Wales, and over 20 years experience working as a software engineer. He has worked in academia, in the defense industry, and in manufacturing. For most of his career, Dr Dyke has been developing for the web. He is currently a senior engineer at Neustar in the Web Performance Management Group.

, , ,