Peering problems aren’t technical issues, but economic ones

Congestion between last mile ISPs and other internet participants such as Cogent or Netflix isn’t the result of technical issues, but business ones, according to a report put out Tuesday by the Measurement Lab Consortium, a group of research, industry, and public interest partners housed at the Open Technology Institute. The report pointed out that these interconnection points — where different networks come together — are congested only in some instances and not in others. However, wherever such congestion occurs, it’s the end consumer who gets the shaft.

The M-Lab report covers peering, the practice where large network or content companies agree to share traffic between networks at various interconnection points. Companies can pay to peer with networks as Netflix did with ISPs, or in most cases, they agree to exchange their traffic freely sharing or having one party take on the minimal cost of servers and equipment needed to make a direct interconnection. Peering is different than buying additional capacity (called transit) on networks owned by companies such as [company]Cogent[/company], [company]XO Communications[/company] or [company]Level 3[/company]. Even major ISPs such as [company]Verizon[/company], [company]Comcast[/company] and [company]AT&T[/company] also offer transit capacity for a fee.

Screen Shot 2014-10-28 at 1.38.42 PM

What’s at stake in the peering debate

The report points out that such congestion occurs primarily between certain providers and during the “peak hours” of between 7pm and 11pm. This is what the FCC and others have defined as internet prime time. The report, which has been a long time coming, is part of a complicated effort to understand how the agreements between various networks that make up the internet affect the end user.

Earlier this year, we covered the issue in depth, including data from M-Labs, as part of [company]Netflix’s[/company] fight to avoid paying direct interconnection fees with the four large U.S. broadband providers. At the height of this fight between Netflix and the big ISPs consumers saw their video streams fail, because there was a bottleneck between Netflix and its transit providers where the Netflix streams hit the ISP network. The result was the user saw their bandwidth available for Netflix shrink to 1 Mbps or even less and their quality diminished. This even happened on YouTube and other video providers such as Hulu and Amazon.

Those issues looked something like this.
Those issues looked something like this.

It was a battle Netflix lost. It now pays Time Warner Cable, AT&T, Verizon and Comcast to peer directly, but it is still fighting the issue with the FCC. Indeed, the FCC is investigating the peering question and gathering data on the technical and business issues that play into the interconnection fights. I’m hoping the agency finds this report helpful as it tries to wade through ISP arguments that basically boil down to: Netflix is responsible for almost a third of the traffic on broadband networks in the U.S. so maybe it should shoulder some of the costs of providing that network.

It’s network neutrality all over again

That’s a fair argument to make, except it ignores that consumers already pay ISPs to provide that network, and the ISPs seem to be able to continue providing service and upgrades even as they continue profiting off their residential bandwidth services. The ISPs also ignore that they appear to be using their duopoly last-mile access to the end consumer as a means to extract payment from companies that want to reach them — something Netflix feels is a violation of network neutrality and even something at which the FCC looks askance (even though it is adamant that interconnection is not a network neutrality issue).

It’s also worth noting, as the Measurement Labs report does, that two providers — Cablevision and Cox — don’t see the degradation at interconnection points that the four largest providers do. Both of those companies accepted Netflix’s content caching boxes as part of its Open Connect service, where Netflix puts equipment inside the ISP’s data centers and delivers the content to it as a way of cutting down on the bandwidth required to deliver content and as a means of ensuring a higher-quality experience for Netflix customers.

Dave Schaeffer, the founder CEO of Cogent Communications and who has been incredibly vocal about ISP tactics in the peering fight, explained that there are eight ISPs worldwide exhibiting “cartel-like behavior” trying to get companies like his to pay for direct peering. The eight companies are Telefonica, France Telecom, Deutsch Telecom, Comcast, Time Warner Cable, CenturyLink, AT&T and Verizon.

Schaeffer said the problem hasn’t actually gone away even after Netflix signed its deal with ISPs. Cogent and others are still feeling pressure — a pressure that it has tried to address directly through negotiations with the ISPs and now through complaints with the FCC.

And that’s what’s most discouraging about the data in the report. It still shows degraded traffic patterns today between middle mile providers such as Level 3 and Cogent in certain areas for the country, even after Netflix capitulated on payments. This shows that Netflix isn’t the only target here, but any content provider paying for transit from middle mile companies. It’s likely some of the larger internet companies are getting hit up for direct paid peering agreements to bypass this manufactured congestion. Schaeffer says he is.

From the report:

[blockquote person=”” attribution=””]As in Atlanta, download throughput for Verizon customers in Chicago begins to decline in June 2013, and experiences the same minor and short-lived increase in March and May 2014. In Chicago, between September 22 2013 and March 2014, download throughput remained consistently less than 10 Mbps, with a low of 4.9 Mbps in February 2014. The performance change that occurred between March and May 2014 led to an increase in download throughput of 5.9 Mbps. Despite this, download throughput to the Chicago sites had yet to exceed 14 Mbps, and the most recent data suggests that download throughput has declined again.[/blockquote]

What’s the solution?

While this data is only that — data — and the report tries hard to stick to technical measurements, there are underlying economics and competitive issues at play here. For example, the cost of buying more servers for creating more interconnections between Cogent and Verizon’s network would cost about $10,000 per 10 gigabit port, said Schaeffer.

“For [all eight ISPs trying to charge for peering] to make problem go away, the one-time capital cost would be less than $10 million,” Schaeffer said. “We have gone as far as saying that for about the 20 percent of internet that travels over Cogent, we would pay 100 percent of our costs and the ISP costs to upgrade. We have pledged to any ISP that we will buy the equipment or pay them to make these upgrades, but even though these eight companies combined spend $100 billion a year in capital expenditures they can’t find a way to make a $10 million problem go away.”

If we can divorce the conversation from one about limited capacity because of technical reasons and cover the real issue — that some ISPs are using their monopoly access to the end consumer as a way to charge content companies money and control what goes on their pipes, we can have the discussion about peering that we need to have.

This report should help give the FCC some of the data it needs to move the conversation in that direction. For consumers or other interested parties, M-Labs also created a site called The Internet Observatory that lets you see how different connections between ISPs and other middle mile providers are handled around the country. M-Labs also makes it data available to all, so you can download it and see if you can come up with different metrics or tidbits about interconnection problems.