An unusual glitch struck Google’s ad serving tool on Wednesday morning, causing blank spaces to appear in place of advertisements on websites like BuzzFeed, Time, Forbes and VentureBeat. The outage means a loss of revenue for hundreds of publishers, and represents the rare failure of a key piece of internet plumbing.
The approximately two-hour failure, which lasted until 10:45 am ET, is likely to have cost Google and publishers millions of dollars.
The ad tool in question is called Doubleclick for Publishers, and is used by a wide range of websites, including Gigaom, to display the ads that ordinarily appear alongside stories like this one.
“DoubleClick for Publishers experienced an outage this morning impacting publishers globally, across their video, display, native and mobile formats. Our team has worked quickly to fix the software bug and it’s now back up and running, so our publisher partners can return to funding their content,” said a Google spokesperson by email.
News of the failure began to circulate on Twitter and on [company]Google[/company] forums around 9:15ET, as publishing and advertising workers began to ask where all the ads had gone.
A person on a Doubleclick forum, however, said the company sent the following message, describing the issue as “critical”:
We’re addressing as a “P0” extremely critical issue across DFP. We don’t have a timeline at this moment but I hope to have a timeline/resolution very soon.
Meanwhile, some wags on Twitter encouraged readers to enjoy the ad-free experience:
https://twitter.com/NatPoulter/status/532548764037566464
The outage affected the many publishers who rely on Doubleclick, but not those like Huffington Post which rely instead on AOL’s ad server technology, or those that use Facebook-owned Atlas.
The failure is reportedly affecting YouTube ads, as well as Google’s AdSense platform.
Other websites affected by the failure included Vox, the Verge, Gawker and MarketWatch.
Here’s a message Google sent to publishers after the incident:
As you are aware, our DoubleClick for Publishers (DFP) ad server experienced a global outage on Wednesday, November 12 from 5:45am to 7:31am PST. This was the first DFP outage of this significance for many years and we take it very seriously. Our team’s first priority was restoring service and making sure the immediate issue would not recur. We are writing now to explain what happened and what we are doing to protect against future incidents.
The details:
| • | The DFP ad server relies on an internal service that began degrading in performance. This caused a cascading failure on DFP ad servers, leading to the outage. |
| • | We designed our systems to gracefully handle performance degradation from dependent services. However, due to a misconfiguration, we were unable to prevent the outage. |
| • | To restore ad serving and prevent cascading failures, we restarted the services by provisioning additional resources. |
| • | We reproduced the failure in a test by degrading the availability of the internal service, proving the misconfiguration caused the cascading failures. We have since rolled out a fix to the configuration globally. |
| • | We are conducting a complete review of all our processes and production configurations to prevent this from happening again. |
As always, our goal is to provide a best-in-class product and the unmatched global scale and reliability you’ve come to expect from Google and DoubleClick. We value your partnership deeply and apologize for the disruption this incident caused to your business.
Best,
Neal Mohan – VP, Display & Video Advertising
Scott Silver – VP, Engineering
This story was updated several times as more information became available.
