Outside of the few content owners that operate their own CDN, third-party CDN vendors deliver nearly every byte of Internet-related information, content, and data that exists. For all intents and purposes, their customers are almost literally everyone in the world.

CDNs provide two essential components for how we consume media today. First, they enable the wide distribution of content using a 1:1 distribution method. Thanks to the Internet and CDNs, content providers can deliver content whenever and wherever it’s requested and get much more information about their viewers (and their preferences) than broadcast TV ever allowed. This means content providers can essentially understand their audience down to a single viewer with the goal of offering individualized experiences so they stay engaged and subscribed.

Second, CDNs provide the scale needed to deliver all this streamed content, which amounts to tens of billions of hours yearly. To handle this level of distribution, CDNs work with Telcos and ISPs to serve streaming media to everyone requesting it, wherever they’re located, in the best possible response time and with the best possible quality. That’s a staggeringly huge lift, and it’s only getting harder as VOD and live streaming grows.

CDN Assumptions: What Content Providers Should Know

A few common assumptions about CDNs have caused the bigger picture to be often misunderstood, and I’m determined to shed more light on these areas. These assumptions might seem small or insignificant, but with increased streaming demand and major shifts like live streaming sports, they could negatively impact the media industry in the long term.

• • Assumption 1: Large CDNs are connected to all ISPs, even remote ones
    While large CDNs are connected to remote ISPs in “theory”, the idea that they are directly connected can be false. Think of the CDN<>ISP ecosystem like an onion. Global CDNs are only directly connected to the handful of larger networks surrounding them. In reality, most ISPs are indirectly connected to large CDNs, sometimes multiple network hops away (in the outer onion “layers”). These ISPs are often smaller and serve less populated subscriber networks – a fact that comes with its own set of problems around upstream capacity and potential impacts on QoE, especially when network traffic spikes.

• • Assumption 2: Large CDNs ensure packets are always delivered quickly and reliably
    While global routes allow content providers to send packets anywhere, they have no control over the efficiency of packet delivery from the CDN to the next connected network – or even the pathways data packets take as they travel across the Internet. Content providers may know that the packet arrived, but how well did it flow to the ISP network further downstream? Did it have to be sent three times because it got dropped due to network congestion?From the somewhat centralized position of a large CDN, it’s difficult to validate and verify every connection to every ISP, especially those located further downstream. Unfortunately, content providers often have little visibility into how quickly or reliably packets reach their destination. Blind spots in content delivery and performance metrics are becoming a big problem that could impact revenue or induce subscriber churn.

• • Assumption 3: Recent funding (e.g., fiber to the home) means we can now deliver higher-quality service in hard-to-reach networks 
    Global initiatives like Project Gigabit in the UK have recently been greenlit to fund new ISPs and AltNets serving broadband and wireless Internet in remote networks. While this is a big step in the right direction, the reality is that managing content delivery just got more complicated. With further-reaching access to fiber, content providers may not realize the backhaul capacity issues that ISPs now face, especially with the surge in live streaming. When network congestion drives consumption upstream (like live sports events often do), everything is ultimately connected to the same large CDN, and every smaller ISP suffers.Some ISPs even have to rebuild portions of their network to handle the scale, which can be costly and disruptive. Scale problems are already complicated, and live streaming with large concurrent viewers is setting up an even bigger battle. Receiving more volume at a higher rate means less time to solve an issue. With subscribers quick to complain on social media and treating contract-free subscriptions as disposable, content providers should carefully consider whether they have the right delivery partners in place.

As the demand for video content and live streaming continues to multiply and consumption becomes ever more individualized, these issues will only get more amplified and complex. In upcoming posts, I’ll expand on these challenges and how content providers can get ahead of them. Here’s a hint: thinking smaller and getting more granular will ultimately support the bigger picture.

Qwilt and Comcast announced a deal that will see Comcast deploy Qwilt’s open caching platform inside its network at hundreds of locations over the rollout, with the first locations going live in the next 1-2 months. The companies are not disclosing the capacity Comcast will deploy within their network or the percentage of total video traffic that Qwilt’s Open Edge platform will deliver across Comcast’s network. However, with the addition of Comcast to Qwilt’s federated CDN strategy, Qwilt says their total on-net coverage of US broadband households will be 55% at the end of this year. While Qwilt has full US regional coverage like other commercial CDNs, its on-net coverage, with many embedded edge delivery nodes, distinguishes its operation and business model.

While Qwilt started as a point solution selling to ISPs, the company has transitioned to becoming its own federated CDN, selling delivery across multiple ISPs directly to content owners. It is starting to compete more with traditional CDN providers like Akamai, Fastly, and Edgio, using its federated CDN model to get a percentage of traffic from customers who use a multi-CDN approach. I have previously highlighted how I have seen Qwilt in the mix for a small percentage of live event traffic from Prime Video, Peacock and other large content distributors. With the Comcast deal giving Qwilt a much larger footprint, the company hopes to get a larger share of customer traffic going forward.

In the federated CDN model, Qwilt shares revenue with ISPs, as they will do in their deal with Comcast. A big advantage that Qwilt has over traditional CDNs is that the ISP provides the hardware—allowing Qwilt to add capacity at a fraction of the cost of competitors since Qwilt does not have to spend capex on edge infrastructure since the service provider owns and operates that part of the network. Comcast claims that by deploying Qwilt within their network, Comcast will create “the most distributed content delivery network (CDN) in the US.” While their wording is too generic, Comcast means most distributed within an ISP environment. Comcast will not have the capacity that Akamai or select other CDNs have in the US. Being widely distributed is crucial to offering a CDN service, but so is capacity, which must go hand-in-hand.

Meta and Deutsche Telekom are having a peering dispute, with the companies ending their interconnect deal. While the two sides are blaming each other, Meta is under no legal obligation to pay Deutsche Telekom for access to their network. Deutsche Telekom is under no obligation to give Meta peering for free. As Deutsche Telekom correctly points out, “Meta alone decides how Meta traffic is routed to Deutsche Telekom’s network.” Meta has switched to using transit providers instead of a direct interconnect to connect to Deutsche Telekom’s network.

In its blog post, Meta says, “given the court ruling concerning the unprecedented and unacceptable fees demanded, we are now routing our network traffic through a third-party transit provider.” Meta doesn’t say how Deutsche Telekom’s fees compare to Meta now paying transit providers to connect to Deutsche Telekom’s network. Deutsche Telekom disclosed that Meta used to pay for access to their network but stopped paying in March of 2021. Why did Meta stop paying? Was there a capacity issue? SLA issue? It’s rare for a platform provider to pay for a direct interconnect to a network and then just stop paying. Is Meta comparing Deutsche Telekom’s fees to what they had for free? Any fee seems “unacceptable” compared to what a company got for three years at no cost.

Meta says Deutsche Telekom “is using its market power to put its subscribers in Germany behind a de facto paywall, potentially restricting their access to internet services that do not pay Deutsche Telekom.” This is not accurate. Deutsche Telekom has not restricted or blocked any of Meta’s apps or services, and Meta knows this, which is why they used the word “potentially.” There is no business reason for Deutsche Telekom to restrict any of Meta’s services, as that would only harm Deutsche Telekom’s customers, who would complain to Deutsche Telekom, not Meta.

I see some in the media saying instead of Meta, “continuing to pay extortionary network fees for a direct connection,” which isn’t accurate. Meta hasn’t been paying anything for the past 3+ years. And what are the fees exactly? Who’s to determine what is considered “extortionary.” I also see some suggestions that Instagram, Facebook, and WhatsApp will “slow to a crawl,” which again is inaccurate. As Deutsche Telekom stated, “the rerouting of data traffic in the night from Tuesday to Wednesday went smoothly.” We see no reports of any Meta apps or platforms having QoS issues, and the media needs to stop suggesting that there are user issues when they do not exist.

While this all boils down to a business dispute between two companies, it’s important to point out that a German court ruled in this contractual dispute between Deutsche Telekom and Meta based on the principles of German contract law. Meta may not like the outcome of the ruling, but it’s the court that decided. Meta is free not to pay Deutsche Telekom and instead move to a transit provider model as they have done.

All of this aside, settlement-free peering and paid interconnects are the basics of how the Internet works so well. I disagree entirely with Deutsche Telekom’s statement that “regulations are needed to ensure the settlement of disputes.” Regulators have no idea how any of this works, and any time they do get involved, it is always behind closed doors. It becomes grandstanding by politicians and lobbying organizations paid by both sides, getting in the middle of the debate. Many step forth with studies and papers implying they are neutral when they get paid by lobbying organizations and have conflicts of interest. The public doesn’t get the details of the contracts or costs, and any shared data is always used to skew the argument without the complete data set being shown.

Regulators won’t fix this, nor should they be allowed to. The biggest problem in this whole discussion is the media. Too many suggest blocking, throttling or slowing down of Meta’s services is taking place when it isn’t. Or speculate, with no proof, that Deutsche Telekom won’t put enough capacity in place with Meta’s transit provider, which again, we know isn’t a problem. Analysts and media members need to stop taking sides and report the facts. All the what-ifs, what could, and what might happen are speculations on their part, and you can’t have a rational discussion and solve business problems focused on theory. Stick to the facts.

Deutsche Telekom Post: Meta is not above the law
https://www.telekom.com/en/company/details/meta-is-not-above-the-law-1079704

Meta post: Why We’re Having to End Our Direct Peering Relationship With Deutsche Telekom
https://about.fb.com/news/2024/09/why-were-having-to-end-our-direct-peering-relationship-with-deutsche-telekom/

For OTT services, the largest content streamers now serve around 40-80 million subscribers in the US every month. Outside of Netflix, they rely on third-party CDNs to deliver all or nearly all of their video. Delivering this content requires a high threshold of HD or better video quality, high bitrates for viewing on large screens, immediate player startup, no buffering, and zero dropped packets during the stream. In other words, a consistently high Quality of Experience (QoE) for end users and that’s magnified even further when streaming live events. On average, for 90-95% of their subscriber base in the US, third-party CDNs provide enough data to measure QoE so that streamers are, for the most part, confident their content is being delivered with an optimal experience. But what about the other 5-10% of their audience?

The Blind Spot Downstream
For subscribers further downstream from where third-party CDNs are deployed, it’s challenging to ensure QoE and almost impossible to validate it. This ‘blind spot’ typically exists in more rural networks, or what many large content providers call “the 1% networks”. In this case, it does not refer to the financial elite but rather the QoE-poor or, more accurately, the QoE-unvalidated. Talking to content distributors, it’s not uncommon for them to tell me they lack visibility into what’s happening in these harder-to-reach 1% networks, mainly due to the inherent function of commercial CDNs. QoE could be good, bad, or somewhere in between, but there’s no telemetry to know. Furthermore, 1% is a figure of speech, not a quantifiable reality; the actual figure could be considerably higher—and many believe it probably is.

In particular, the continued viewership growth in live sports streaming can increase traffic by 25-50% or more for some networks, requiring sufficient capacity to ease bandwidth congestion. For commercial CDNs and ISPs, these traffic spikes can be more easily absorbed, as capacity can be spread over a larger area. However, for the smaller ISPs serving 1% of networks, capacity is often constricted unless extra capacity is purchased in advance, called an RSVP fee, and often at a premium, and that’s if capacity is even available.

Then there’s the content delivery blind spot. With live streaming, QoE metrics are often received directly from the player in real-time. Content streamers can easily detect content delivery and streaming performance issues for commercial CDN and ISP networks. For the harder-to-reach 1% networks that are much further downstream, however, tracking content delivery performance is difficult, if not impossible. This means subscribers can easily suffer from buffering, slow startup times, dropped packets, and low bitrates – when they want to stream content the most.

Live streaming from the NHL, NBA, and US Open sporting events has quickly become the most-streamed content across digital platforms, with the NFL dominating TV viewership in the US. When QoE is suboptimal during the big game, the blind spot in a content delivery pipeline suddenly becomes a glaring problem for 1% of network subscribers. This is a common problem for ISPs who receive these complaints from viewers expecting a better experience.

Do the Math
While I’ve called out the 1% of viewers being impacted, in reality, it’s more. One percent of subscribers may sound like a minimal amount, and not much to be concerned about, but content owners will want to think again. If you aim for the best but plan for the worst and then do the math, you’ll realize that 1% is significant. For a content streamer with 40 million subscribers, one percent is 400,000 subs. At $7/month for an average plan, that’s $2.8 million in monthly subscription revenue or $33 million annually.

Moreover, 1% is just an idiom referring to harder-to-reach networks generally. That 1% makes up 5-10% of an OTT platform’s subscriber base when it comes to the blind spot of content delivery and streaming performance. With 5% of network subscribers potentially experiencing poor QoE, a content streamer with 40 million subscribers is now looking at 2 million subscribers and $14 million in MRR—or $168 million annually.

Another angle to consider is customer churn and acquisition costs (which streamers don’t disclose, even to Wall Street). These costs vary widely across organizations; Even without knowing the exact numbers, keeping existing customers is almost always more economical than acquiring new ones and a much easier lift. As for winning back churned customers? That cost just sounds steep. Suffice it to say that if you can’t validate the QoE for your hard-to-reach subscribers, OTT platforms and broadcasters are risking customer churn and lost revenue. They are risking brand reputation, media partnerships, and exclusive streaming rights for the content people want.

QoE in 1% networks has been obscured and underrepresented for far too long, and there’s a lot more at stake than most people think. It’s just that no one seems to be shedding much light on it, let alone doing the math. This gaping hole has only gotten wider for as long as I’ve been in the streaming media industry. Between more people leaving big cities to work remotely or cut living costs and increasing demand for streaming video (especially with live sports), this is a growing problem. Having a blind spot in a content delivery pipeline and potentially delivering suboptimal QoE shouldn’t be acceptable anywhere, especially nowadays, and it deserves more attention.

The last-mile CDN
Part of the problem is that no broadcaster or OTT platform will disclose what percentage of their users had a good experience after live events. Nor will they discuss max bitrates, average startup times, average viewing times, rebuffering or their CDN strategy’s impact on their viewers. As a consultant, I’ve worked on some of these large live events for broadcasters, having access to their monitoring platforms during the events and have seen the problem firsthand. And while NDAs keep me from giving out numbers, a larger percentage of users are impacted than most would think. Talking off-the-record to many of the live event teams at any of the large OTT platforms confirms the scale of the impact.

This market problem was why I profiled Canada-based Netskrt a few months ago after it raised about $7.2 million in its Series A round. The company focuses on delivering video to remote and rural communities for specific video applications with hard-to-reach subscribers to help smaller ISPs manage live event traffic. Effectively, they provide a last-mile CDN for ensuring content delivery and streaming performance for 1% networks. Plus, it’s way more cost-effective than building your own cache, and it’s certainly smarter than ignoring your QoE blind spot. Netskrt says they plan to discuss rural network-related challenges in the coming months and release some QoE-related data to debunk some common misconceptions. That data will be interesting to see, as well as how well the company adds another 15 Tbps to its US footprint and expands it to include Brazil, the UK and Italy. Expect to see me blog more about this topic when Netskrt provides some data.

The future of content delivery for broadcasters has changed dramatically over the past few years and will continue to do so, especially for live events. Some have predicted that in the next five years, broadcasters will need to deliver TV-sized prime-time audiences, with streaming media technology being the primary means of distribution.

This means that current peak streaming audiences of about 3-4 million in a country with a population of 65 million will transform into peak audiences of 20-25 million for the same event, using the viewer figures published about the Euros 2024 Football championships as an example. This means that streaming services’ capacity, quality, and security requirements will become dramatically more important for broadcasters in the years ahead.

While I don’t see streaming concurrent viewership of 20-25 million as the norm, at a regional level, we are starting to see some very large concurrent audiences for one-off live events and specific sports matches. DAZN is a good example of a platform that sees large concurrent audiences for Serie A and Spanish LaLiga, where the audiences are large, and the demand for the content is strong.

As part of DAZN’s multi-CDN distribution approach, working with MainStreaming, DAZN built and operates a private CDN-as-a-Service, deployed and managed in close partnership with the largest ISPs in DAZN’s core markets where DAZN Edge is deployed. Not every streaming platform or broadcaster can, or should, build their own CDN; for most, it doesn’t make sense. But in cases like DAZN’s, especially at a regional level, a private CDN-as-a-Service can work well at the core of a multi-CDN strategy.

From a QoE perspective, DAZN says the private CDN has proven its worth, fine-tuning configurations and deployment plans with MainStreaming to ensure the best possible performance to support regional spikes from large audiences tuning in for specific games. The chart above, published in 2023, shows why DAZN’s private CDN at the core of a multi-CDN model provides the base performance it requires.

The internet is constantly in flux as demand on networks fluctuates without notice. Network connections change without warning as ISPs adapt their networks. Yet this ability to change continuously creates different network paths for streaming video to traverse. For example, BGP Routing tables for IPv6 connections (see below) show a general growth trend over time. Still, connection counts have gone down and up quite dramatically, immediately changing how video traverses the internet.

Unlike broadcast, which has standards about QoE, in the streaming industry, there are no agreed-upon definitions of what HD or 4K streaming even means, let alone the term “Broadcast-grade” streaming. I like how MainStreaming defines the term as six key components, and I’m curious to hear in the comments session how others define it:

• Deliver media with consistently high quality and low latency; and
• Scale to many millions of viewers; and
• Achieve predictable delivery costs with economies of scale; and
• Take real-time QoS actions from real-time QoE analytics; and
• Protect against Piracy through CDN-embedded controls; and
• Deliver in Ultra-Low Latency to meet or beat an App’s speed

Broadcast-grade streaming is necessary for broadcasters and streaming platforms to protect their business as they move to a more streaming-first strategy. While this brings new opportunities to engage audiences better and increase revenues, it also brings extra threats to content security, advertising revenue, subscription revenue, and brand reputation. Private CDN platforms will need to adapt to address all those concerns and, in many cases, will need to integrate with other vendors in the video workflow who are specialists in addressing some of those problems.

Edgio announced that it has voluntarily filed for Chapter 11 bankruptcy, and I expect the company to be delisted from the NASDAQ by the end of the day or shortly after. (Updated: Edgio was delisted on Monday September 18) This news does not mean the company is going out of business. Edgio will receive approximately $15.6 million in financing from Lynrock that, following approval by the Court, is expected to ensure that Edgio continues to operate like it is today and throughout the sale process and Chapter 11 Cases.

The purpose of the filing is so that Edgio can sell off part of the business it no longer wants or the entire company, which should allow for the continued operation of the Company’s business under new ownership. It wouldn’t be an asset sale but a business unit sale or the entire company. There is a 40-day bid procedure process, and once all bids are in, the court will decide which is best for the company.

Edgio says they have recently engaged in discussions with several parties that could be interested in acquiring all or part of the Company’s businesses and assets, and by using the Court-supervised sale process, they seek to get the highest or best bid for those assets. As part of this process, Edgio has entered into a “stalking horse” asset purchase agreement with its primary lender Lynrock, which has agreed to acquire assets of the Company through a credit bid for $110 million of the existing secured debt held by Lynrock.

Whatever assets might be sold, Edgio says it is targeting the sale process to be completed in approximately 80 days, if not sooner.

Note: I have never bought, sold, or traded shares in any public CDN, and even in my managed portfolios, Akamai, Fastly, Cloudflare, and Edgio are excluded.

Here’s a list of streaming viewership numbers from NBCU for the Olympics across Peacock and NBCU Digital platforms, averaging 4-5 million viewers. NBCU clarified that these numbers are AMA (Average Minute Audience), even though their releases don’t say that.

• August 3: coverage was streamed by 𝟰.𝟰 𝗺𝗶𝗹𝗹𝗶𝗼𝗻 𝘃𝗶𝗲𝘄𝗲𝗿𝘀 on Peacock and NBCU Digital platforms
• August 1: coverage was streamed by 𝟰.𝟰 𝗺𝗶𝗹𝗹𝗶𝗼𝗻 𝘃𝗶𝗲𝘄𝗲𝗿𝘀 on Peacock and NBCU Digital platforms
• July 30: coverage was streamed by 𝟱.𝟬 𝗺𝗶𝗹𝗹𝗶𝗼𝗻 𝘃𝗶𝗲𝘄𝗲𝗿𝘀 on Peacock and NBCU Digital platforms
• July 29: coverage was streamed by 𝟰.𝟱 𝗺𝗶𝗹𝗹𝗶𝗼𝗻 𝘃𝗶𝗲𝘄𝗲𝗿𝘀 on Peacock and NBCU Digital platforms
• July 28: coverage was streamed by 𝟲.𝟬 𝗺𝗶𝗹𝗹𝗶𝗼𝗻 𝘃𝗶𝗲𝘄𝗲𝗿𝘀 on Peacock and NBCU Digital platforms
• July 27: coverage was streamed by 𝟰.𝟳 𝗺𝗶𝗹𝗹𝗶𝗼𝗻 𝘃𝗶𝗲𝘄𝗲𝗿𝘀 on Peacock and NBCU Digital platforms
• July 26: coverage was streamed by more than 𝟮.𝟱 𝗺𝗶𝗹𝗹𝗶𝗼𝗻 𝘃𝗶𝗲𝘄𝗲𝗿𝘀 on Peacock and NBCU Digital platforms

The above language is taken directly from NBC Sports press releases.