Netflix (s NFLX) had a team of four or five people working on its OpenConnect caching appliance, the company’s cloud architect Adrian Cockcroft revealed at GigaOM’s at GigaOM’s Structure conference in San Francisco Thursday, and the team went from drawing up first plans to deploying it to major ISPs in just a few months.
OpenConnect boxes are being used to cache Netflix content within an ISP’s data center, bringing it closer to the end user and thus improving the quality of the service. Cockroft said that Netflix opted to build its own systems for this kind of caching because it was much cheaper than buying it from a major hardware vendor. Part of the reason for that is that the legacy vendors build multipurpose systems that have to be tested and optimized for many different use cases. “We wanted a box that would do one thing,” Cockroft said.
Netflix’s OpenConnect development was directly influenced by Backblaze, whose CEO Gleb Budman joined Cockroft on stage to talk his company’s backup business. Backblaze started to offer online backup services five or six years ago, and Budman said that the company early on realized that AWS simply wasn’t an efficient way to go, which is why the company set out to build its own storage infrastructure.
Today, Backblaze uses more of 60 petabytes of storage, and the company is adding 3 additional petabytes a month. Budman estimated that the company has saved more than $100 million dollars by building its own storage hardware instead of using AWS over the years.
But those savings wouldn’t have been possible if Backblaze had just bought its hardware from established vendors. Legacy suppliers were charging $3000 for boxes with a terabyte of storage, while disc drives with a terabyte only cost $100. That’s why Backblaze designed its own low-cost storage system, which it eventually open sourced.
Cockroft said that this move had a big influence on Netflix’s decision to build its own caching boxes, but he also said that it makes sense for a company like Netflix to use AWS for most things other than caching. Netflix has been running most of its business off the cloud for some time and has become Amazon’s (s AMZN) biggest customer. The humble beginnings for that move were when the service ran into the good problem of growing too quickly. “We ran out of disc space in the data center,” remembered Cockroft.
Check out the rest of our Structure 2013 live coverage here, and a video embed of the session follows below:
[protected-iframe id=”0daf78e48eb7fbd3806b7e9349f7b32a-14960843-61002135″ info=”http://new.livestream.com/accounts/74987/events/2117818/videos/22087010/player?autoPlay=false&height=360&mute=false&width=640″ width=”640″ height=”360″ frameborder=”0″ scrolling=”no”]
This post was updated at 10:08pm to correct the spelling of Backblaze.
A transcription of the video follows on the next page
