Why technology and content are inseparable at Netflix

Neil Hunt is the chief product officer at Netflix, and his job entails a lot more than it might sound like. The end product at Netflix is the video streaming through our iPads or smart televisions, but what we’re watching and we’re seeing it is result of a lot of work.

[company]Netflix[/company] is well known for the algorithms that recommend content to viewers, for example, but less is known about its resilient cloud computing architecture and the myriad considerations that come along with delivering and pricing new types of video formats. In a recent interview, Hunt explained how Netflix approaches all of these things and where it sees opportunities to improve.

First things first: The recommendations

Believe it or not, Hunt thinks the importance of Netflix’s recommendation engine is actually underestimated. Customers watch about 7 billion hours of video every quarter, he explained, which breaks down into about 150 million events per day when someone actively chooses to watch something.

“If you could make just a portion of those, say 10 percent, 15 million viewing events a day, more productive … you could quickly influence the retention behavior of a large number of customers and generate hundreds of millions of dollars of increase in lifetime value from our existing members and not having to replace them at some cost with new members,” he said.

And Netflix is just scratching the surface on improving productivity. Hunt said the company carried out about 300 A/B tests in the past year, many of them involving recommendations, and nearly half of them resulted in users clearly preferring one option over the other.

Neil Hunt.
Neil Hunt.

“Think of it as, roughly, every second or third day there is a measurable improvement in the quality of what we deliver to our customers. That says to me that with that much progress being made right now we’re not near the end,” he said, noting that each improvement could potentially result in tens or even hundreds of millions of dollars for the company. “When we start getting one test out of ten as a success then I know that, well, we’ve tried all the interesting ideas and there is not much left to go and maybe we should not invest so much in it.”

In fact, Netflix is still learning things about how users’ star ratings work, including that ratings given immediately after a movie tend to be more accurate than ratings fired off 50 at a time. And that stars aren’t the be-all, end-all the company once thought they were, in part because people tend to conflate quality with enjoyment.

Hotel Rwanda or Schindler’s List is perhaps a super important movie that deserves a five-star rating, but it’s not necessarily something that you want to sit down and watch after a busy day at work,” Hunt explained.

[pullquote person=”” attribution=””]”If you run into that and you don’t do anything, then it’s relatively safe to say you probably fell asleep or walked away and left the machine.”[/pullquote]

So Netflix now gives a lot more weight to customers’ actual viewing behavior. It can learn a lot based on what people watch next, and even what they search for and end up watching when Netflix doesn’t have the target movie in its library. Rewinding suggests someone is engaged with a movie but sitting through a movie without ever touching the remote might suggest they’re not paying attention.

Think about that warning, which Netflix called the autoplay blocker, that comes up when you’re in the middle of binge-watching a series and haven’t done anything but stare at the screen for three hours.

“If you run into that and you don’t do anything, then it’s relatively safe to say you probably fell asleep or walked away and left the machine. Maybe we shouldn’t count that as a big part of the signal,” Hunt said. “On the other hand, if you run into that blocker and you say, ‘I’m still here’ and you watch another two episodes, then clearly it was a very motivating thing that kept you engaged.”


Building a data organization that works, and works with business

Hunt said Netflix separates the data experts it hires into three separate teams: the folks who build algorithms and do data science for the whole business; the folks who focus solely on algorithms for content discovery; and the engineers, who scale all those algorithms to run at Netflix’s scale. However, he added, the walls between the teams are fairly porous, so good ideas about any aspect make their way through regardless where they were thought up.

But there is a pretty clear distinction between the product teams in Los Gatos, California, and the content buyers down in Los Angeles. “The contract between us, roughly, is whatever they buy we figure out how to get the most value out of it by putting the right stuff in front of the right people,” Hunt said.

This being Netflix, though, Hunt’s teams will still deliver the people buying and creating content “tremendous” amounts of data about what shows are likely to be popular with which audiences, how many hours they might watch and how much bang Netflix can expect to gets for its buck.

Running it all in the cloud, for better or worse (but mostly better)

Netflix famously hosts its streaming infrastructure on the Amazon Web Services cloud (something we talked about more in depth on our podcast this week, with two senior members of its cloud tech team) and Hunt said that in some sense, that infrastructure is the product. It’s the part that’s responsible for delivering the features everyone else develops to all of Netflix’s users, whenever they press a button.

“It’s not that hard to write an recommendation algorithm that generates recommendations for one person overnight,” he said. “[It’s] much more difficult to be able to deliver 150 million choices a day, each based on a selection of how to prioritize a library of 10,000 titles.”

The company aims for 99.99 percent availability, which translates into a few minutes per month of downtime, but Hunt acknowledged it’s not there yet. Despite all the hard work Netflix’s cloud architecture team has done building resiliency and redundancy into its AWS environment — see, for example, its “Simian Army” set of tools that tests the infrastructure against various types of failures — there’s still more it can do.

A high-level diagram of a new Netflix failover system called Dynomite.
A high-level diagram of a new Netflix failover system called Dynomite.

And while Hunt said admitted he’s sometimes nervous about the AWS cloud, he’s been around enough to know that you can never do away with third-party tools. “It doesn’t matter whether you lease the equipment or buy the equipment, somebody else has a big piece of it,” he said.

In fact, Netflix’s biggest outage wasn’t the infamous Christmas Eve outage in 2012; it was in 2008, when Netflix was running its own servers and a disk upgrade corrupted the databases, Hunt said.

One key to embracing the cloud is to put it in context. At least with Amazon, Hunt added, there’s somebody on the other side trying just as hard to fix things. When there was a Xen hypervisor bug in October that required rebooting 60,000 of Netflix’s virtual servers within AWS, a few people stood back to monitor it but most people went to the Netflix 50-million-customer party and didn’t have to worry about it.

[pullquote person=”” attribution=””]”We pay about twice as much for our infrastructure through Amazon as we did back in 2008 when we ran all our own infrastructure, but we’re about 20 times bigger.”[/pullquote]

“I think the other key piece here is that we don’t depend on Amazon’s infrastructure being infallible,” Hunt said. Unless there’s some catastrophic failure, Netflix views most most problems as design failures on its end rather than as problems with Amazon’s cloud.

Of course, there’s more to cloud computing than just worrying about downtime — it also brings some very noticeable benefits. Hunt said Netflix can test whatever ideas it has essentially without worrying about cost or deployment time. It can, for example, test two new recommendation algorithms in full production, shut down the less-effective one, and not worry about too many sunk costs or extra servers.

“We pay about twice as much for our infrastructure through Amazon as we did back in 2008 when we ran all our own infrastructure, but we’re about 20 times bigger,” Hunt said. “That’s a tenfold efficiency increase.”

Amazon Web Services' "Cloud Heroes." Adrian Cockroft used to head up the Netflix cloud team, and Jeremy Edberg does now.
Amazon Web Services’ “Cloud Heroes.” Adrian Cockroft used to head up the Netflix cloud team, and Jeremy Edberg now works there.

The economics of 4K and beyond

[pullquote person=”” attribution=””]”We have unique control over all pieces of pipeline to be able to put 4K in somebody’s living room screen.”[/pullquote]

If you think Netflix is a big company now, Hunt suggests we wait and see what’s coming next, especially as consumers start buying new televisions and want the content to go along with them. He thinks Netflix is a unique position to do what traditional purveyors — cable companies and big-box stores like Best Buy — cannot:

[blockquote person=”” attribution=””]”There’s an important observation that if you’re a television cable channel or a broadcaster, or if you’re delivering media on a Blu-ray disc, and you want to embrace a new format like 4K or ultra-high def, you need to allocate new channels to it because you can’t cut off all the old users, they’re not going to upgrade overnight. That leads to either a bandwidth problem on your cable or a channel allocation and spectrum for the broadcaster, or a shelf space problem for the Blu-ray distributor. All of which impede the adoption of a new standard.

“But internet delivery bypasses some of that. We can, since we’re delivering one to one, deliver 4K for the 1 percent of people who have upgraded their televisions and have a 4K smart TV, without subtracting or taking away at all from the rest of the population.”[/blockquote]

And Netflix can commission a new show, film it in whatever formats it pleases — 4K (aka ultra-high definition) or, next year, maybe high dynamic range — and then deliver it over the internet onto an app it built for a specific smart TV.

“We have unique control over all pieces of pipeline to be able to put 4K in somebody’s living room screen,” Hunt said.

Netflix shot, and streams, "House of Cards" in 4K.
Netflix shot, and streams, “House of Cards” in 4K.

Looking further into the future, Hunt explained a pet project of his that involves working with television manufacturers on a way to let the Netflix video file automatically calibrate picture settings along with a film director’s artistic intent. Most sets are tuned way too bright, and it’s even worse if they’re set to “vivid” or “game” modes. On showroom floors, he noted, sets are often tuned not for the best picture quality but to stand out more.

“That’s a very interesting system problem in the sense that we’re dealing with not just our technology, but the technology of a dozen different television manufacturers,” Hunt said.

As for the extra costs Netflix has announced it will pass along to consumers of these higher-quality video formats, Hunt noted that the $3 a month increase recently put in place for ultra HD isn’t that much different than the cost of upgrading to high-definition from standard definition on some competitive services, and is significantly less when broken down by viewing hours. He also thinks it’s fair.

“It’s a pretty nominal charge for somebody who has got a 25-megabit connection and has just bought a thousand-dollar television set,” he said. “It’s more about making sure that we’re not loading the costs of and bandwidth and storage and all the R&D that goes into it on the student who is trying to watch on the laptop in their dorm room.”