New Data Questions Netflix’s Assertion That ISPs Are At Fault For Poor Quality
In the Netflix versus ISPs peering dispute, there are a lot of opinions and debate around who’s at fault for letting some peering points degrade and who should be responsible for upgrading them. To date, many are having a hard time separating facts from opinions because Netflix and the ISPs haven’t released any concrete data to back up their claims. In most industries, if one company accused another of doing something wrong, it would be expected that the company making the claim would back up their position with detailed data that proves their point and leaves little doubt as to who’s responsible for the problem. Netflix has yet to do that.
Most seem to be giving Netflix a pass, with very few demanding real transparency into what’s taking place, or changed, that degraded Netflix performance back in September 2013. No company should try and force us to take their word for it, they should simply make the data public and let us decide on our own. Netflix says they are bringing transparency to the debate, but they are doing the opposite by using vague and high level terms with no definition. To date, Netflix has yet to set forth any details on how they want the current business models to change, how it should be regulated, what they consider “strong” net neutrality or even submitted a proposal to the FCC.
The best example of this is how Netflix’s player recently gave out messages saying that Verizon was at fault regarding quality issues, but then when challenged by Verizon to back up their claim, Netflix announced they would discontinue showing these messages on June 16th. Originally Netflix said these messages would be rolling out in a phased deployment on all networks, but in their blog post yesterday, they now say these messages were just a “test”. To me, it looks like Netflix simply created noise in the market, again with no data, and then when pressured by Verizon to prove their case, Netflix instead decided to stop sending the messages and not release any details. Why? If the problem lies within Verizon, Netflix should let us see the data that shows this and stand behind it. Why back down if they have the data to show where the problem is coming from?
This is just another example of many where all sides simply point the finger at each other and say it’s the other guys fault, but then provide zero details to back up their claims. However, that may change soon as Netflix will likely publish network and performance graphs around a peering event, taking place in DC on June 18th, to bolster their argument. At the same time, some ISPs are actively working to release some data to the market, like the internal chart below [Figure 2] that I received from a major U.S. broadband provider, which gives us some more visibility into what’s taking place inside an ISPs network.
Netflix’s accusation is that ISPs have purposely congested their peering points in order to specifically degrade the Netflix service. What Netflix has failed to be transparent about is that Netflix has always paid to deliver their traffic. CDNs like Akamai, Limelight and Level 3 successfully managed the majority of all of Netflix’s video and were responsible for Netflix customer performance. Each of these companies successfully delivered Netflix via all the same transit paths and business relationships equally available to Netflix today. When Netflix took over the routing controls for their video traffic with their own CDN Open Connect, customer performance began to suffer as highlighted in Netflix’s own data that they shared with the Washington Post. I added a red circle to the chart to show when the Netflix changes took place and the impact to customer performance by ISPs.
Figure 1.
While Netflix was able to convince smaller regional ISPs to voluntarily offer settlement free peering, most large ISPs maintain national/international infrastructures, which require peering policies and consistent business practices to ensure fair and equal treatment of traffic. For the providers Netflix did not qualify for peering, Netflix moved their traffic onto very specific Internet paths that were not capable of handling their massive load and caused the congestion that impacted customers (as highlighted in Figure 2 below). In other words, if Netflix receives free peering, ISP customers receive good performance and high rankings and blogging praise from Netflix. But if Netflix does not receive free peering, ISP customers do not receive good performance and get low rankings and shame from Netflix.
It was Netflix that specifically chose transit paths to those ISPs who refused to give it free peering that it knew (and measured) were not capable of handling an increase in load. In some cases I was told by ISPs that traffic levels increased by 500% in only a few months where normal Internet growth with these same peers was less than 20-30% across an entire year. These ISPs’ customers did not request traffic to be served from poorly performing paths. Netflix chose to create, and use, paths that they knew were congested, simply because they were cheaper than using paths that were less congested. While some may not like that decision, Netflix is running a business and like all businesses, cost is a factor in a lot of decisions. I’m fine with Netflix having to make tradeoffs between quality and cost, but it’s not true that 100% of every path going into Comcast was “congested”.
Some of the many other transit providers I have spoken with confirm this, saying that they could have handled incremental Netflix traffic into Comcast, but that it would have been more expensive than Cogent, which was Netflix’s primary transit provider at the time. Even Cogent would not deny they were the cheapest transit provider of all the ones Netflix was using, but as we’ve all learned, cheap does not guarantee quality. During this same time, Netflix was still using other third-party CDNs for some of their video delivery. These CDNs were delivering the same Netflix service at the same time, to the same locations and with good quality. That is why some customers said their Netflix video was working great, while others said it was buffering and it is also why if some customers used VPNs, their performance improved. Netflix had control over who to give good service to and who to degrade, as shown in this chart below, from a major U.S. broadband provider:
Figure 2: Major U.S. broadband provider
Red is Netflix Video Bitrate delivered via a 3rd Party CDN
Blue is Netflix Video Bitrate delivered via Netflix internal CDN
When Netflix delivered video through similar sized third Party CDNs, customers received a consistent HD video stream 24 hours a day and from the CDNs I have spoken to, none of them had any problems getting Netflix’s traffic to the ISPs. When Netflix sent the video stream through their own CDN using their used newly congested transit paths, prime time viewing based on these decisions had buffering and low quality video.
Amongst those in the industry, there is a widely used term “traffic manipulation” and is “one of the most clever and devious of all the [negotiation] tactics” as described in The Peering Playbook. As the playbook details, the problems this traffic manipulation strategy introduces are three-fold. It negatively impacts customers; the levels of Netflix traffic also impacts other services sharing the same path; and if the “traffic manipulation” tactic is successful by Netflix or any other large CDN provider, it will be repeated by others creating further instability on the Internet.
Netflix’s point of view is that the Internet has changed a lot since many of these paid interconnect deals were done ten years ago. Today only a few ISPs control the vast majority of the market in the U.S. and Netflix would like to see peering and interconnect business models change. While that’s a fine argument to make, Netflix has yet to deliver any proposal or suggest an alternative other than to say it should be free. Netflix only confuses the discussion by involving “net-neutrality” in the debate and opportunistically point fingers at specific ISPs like Verizon and Comcast. But, Netflix has always had all the same delivery options as every major CDN and video provider on the planet. Many of these CDNs and large content sources have large volumes of traffic as Netflix and able to deliver Netflix and other Internet services with high quality service for their customers. The transmission decisions that Netflix makes are just as suspect to what is impacting their customers’ performance. Putting all the fault on ISPs is not accurate as both sides share the blame in not being able to make this work.
I’ve also heard from some that Netflix has told them that they have self-limited their transit decisions to only ISPs without residential access customers. Most CDNs use ISPs like AT&T, Sprint, Centurylink, Verizon and Comcast as transit options to reach the entire Internet, including each other’s network. Akamai and Limelight for example are connected to every Tier 1 provider and most Tier 2’s to deliver their service. They do not impose these business limitations in order to ensure they deliver services with high quality, so why is Netflix? These are not “peering issues”, these are first mile network decisions that Netflix is 100% in control of.
By Netflix limiting themselves to only Cogent, Level 3 and smaller International ISPs, they could find themselves buying capacity that is just not available. “Just upgrade your peering links” is not always the answer due to Internet peering policies as well as concerns that after investing capital, Netflix will just move terabits of traffic again in a few months making the problem start all over. What if Comcast gave Cogent all the capacity it wanted and footed the bill for it, and then the next year, Netflix moved away from Cogent and used Tata instead? Now Comcast would be forced to have to give Tata all the capacity they wanted and this change could happen every year. In fact, over a 3-4 year period, Netflix moved some or all of their video traffic amongst multiple CDNs including Akamai, Limelight and Level 3 and in some cases, due to lower pricing, left one provider only to come back to them a year or two later.
Netflix dominates close to 70% of the long form streaming traffic which is 10x their nearest competitor. This market dominance allows Netflix to use their massive traffic controls in ways to force or demand special free peering privileges from ISPs rather than continuing to include these market based delivery costs in their service as others do. These costs that Netflix is objecting to have always been a cost of doing business for large CDNs in the U.S. That’s why to date, no other large CDN like YouTube, Apple, Microsoft, Amazon, Akamai or Yahoo! has come out publicly and backed Netflix’s position. Some ISPs, like Google Fiber, have said they think interconnects should be free, but that’s not the CDN portion of Google’s business, which for years, has done some paid interconnect deals.
Netflix has always paid a portion of their traffic delivery cost just like all of Netflix competitors large and small. Netflix is unhappy that they are being held to the same Internet policies as every other player. With great power comes great responsibility, and if Netflix will be operating one of the largest CDNs in the world, they must start behaving like one of the largest CDNs of the world. If they think the current model is broken or feel that market dynamics have changed enough to where new models needs to be put in place, that’s a fair argument. But you can’t make that argument and expect any change when you don’t put forth any kind of proposal. Netflix says they want changes that are “fair” and rules that are “strong”, but those words don’t mean anything without details.
Netflix hasn’t been clear with their arguments. First it was really about transit, but now they are saying it’s about “terminating access monopolies,” broadband competition and such. Problem is that is a much different issue and they’re having a hard time making that case and tying it to why they must have this or that remedy to lower their own costs. If broadband competition is now their issue, then it clearly is unrelated to net neutrality or the Comcast and TWC merger since the companies don’t overlap. There are other ways to go about addressing broadband competition constructively without bad-mouthing ISPs and confusing the public and throwing out made-up regulatory remedies like Title II or strong net neutrality.
If you talk to the major transit providers, CDNs, ISPs and the companies tied up in this debate, many will detail what has been going on behind the scenes with traffic manipulation and only using the data that makes them look good. There is more to it than that, and it’s one of the main reasons why those involved won’t detail what’s really taking place, where the problem lies, what their business motivations are for some of the decisions they make and the impact these decisions have on their bottom line. Remember, Netflix locked in a fixed rate with Comcast, for more than five years, for nearly a third of their traffic, which gives them a clear advantage over any other OTT competitor. So lets not forget the positive impact these interconnect deals are having on Netflix’s bottom line. Cheaper costs, lower churn, better quality video and the longest length contract, which I have come across, in the industry.
If Netflix really believes broadband competition is an issue, why make enemies with the companies who you can partner with to address it? To me, it all seems like an irrational strategy, especially when some of the arguments Netflix makes, don’t align with what some transit providers, CDNs and data from within ISPs shows. If Netflix wants real transparency, then they need to come out and put their cards on the table and show us what they have. It’s easy for anyone to complain and blame the other guy, but just because you might not like your ISP, think your cable bill is too high or have some other reason not to like your ISP, that’s not a reason to give Netflix a free pass and not demand they give us the transparency that they themselves keep saying they want.