Facts matter, except apparently in the aftermath of Netflix’s boxing event. This was Netflix’s 8th 9th live event (open to correction), not their first. Everyone is an expert in CDN and can “fix” Netflix’s live streaming, even though they don’t know the problems. Multiple vendors claim their cloud/edge/P2P/”insert anything here” solution could have solved the problem when some vendors are in beta with zero deployments. Many people share and compare numbers that are completely made up but state them as fact.

Having a fruitful conversation on the topic is hard since many refuse to take the time to educate themselves on the real numbers of past events. The volume of inaccurate posts I’ve seen on LinkedIn wastes people’s time. Stop posting made-up numbers. Here are a few points I keep seeing that need to be corrected:

• FALSE: “It was Netflix’s first live event, so it will take them time to get it right.” This was Netflix’s 8th 9th live streaming event. (Love is Blind reunion, Netflix Cup, Chris Rock Special, 30th annual Screen Actors Guild Awards, UDUM 2023, The Netflix Slam, Tom Brady Roast, Joe Rogan Special)
• FALSE: “The NFL games on Christmas will need capacity for 100+/110M concurrent streams”. The Las Vegas Raiders and Kansas City Chiefs drew an average audience of more than 29 million viewers on CBS — which made it the most-watched Christmas Day game in 34 years. Even with Beyonce as the halftime show, Netflix is not getting 110M concurrent streams.
• UNKNOWN: “Only 0.1%/0.15%/1%/5% of the audience had a bad stream.” No one knows the percentage of overall users that had bad QoS, and people are pushing out numbers with ZERO sources, public or private. Using reports from downdetector is NOT a valid source!!!
• FALSE: “Netflix spent ~$50M on the event.” Based on what the fighters said, the purse alone was more than $80M combined. Add marketing and production costs, and Netflix spent well over $100M to produce the event. I don’t know if they shared in the revenue from the venue ticket sales, and I have found no source of Netflix commenting on the topic. I also don’t know what Netflix made on sponsorships.
• FALSE: “Akamai had major problems delivering the videos.” Netflix used its Open Connect infrastructure exclusively, and no third-party CDN was used. Some said, “If Netflix relied exclusively on its Open Connect CDN…” Why is anyone guessing? Do some traceroutes and talk to the vendors. All third-party CDNs were very open when asked and said they had no involvement in delivering the stream.
• FALSE: “Netflix supplemented its own CDN with vendor CDNs for all their overflow traffic.” See above.
• FALSE: “Netflix failed to test its capacity load properly.” You can’t simulate 65M concurrent streams in any real-world testing environment. Many want to point to “capacity” with ISPs, which I know, in many cases, was NOT the problem, as the ISPs told me.
• FALSE: “The event can only be compared to the Super Bowl.” Super Bowl streaming viewership peaked at 8.5M last year; on TV, it peaked at 126.3M in 2017, and none of those 126M viewers had QoS issues.
• FALSE: “Netflix has 8,000+ servers.” According to the last number they released, Netflix has close to 20,000 servers deployed across 175+ countries. In 2023, a half-rack of eight servers could deliver 200,000 concurrent streams. (Netflix did not say if that was VOD or live streams)
• UNKNOWN: “Estimates put sign-ups from this event at 10% / Netflix signed up millions of new users.” Based on what source? 10% of what number?

For those making up numbers and publishing them without sources, and you have the title of “Research Director” or “Analyst,” shame on you. Playing fast and loose with numbers is not acceptable. All my numbers in the post are sourced from public blog posts and/or press releases from Netflix, CBS, and Paramount.

In Edgio’s bankruptcy asset auction, Akamai won the bid for Edgio’s “Apps and Security Business Assets and Network Assets.” Radware’s backup bid is for the Apps and Security Business Assets segment but not the Network Assets segment. The bankruptcy Judge will hold a hearing on November 25th to consider the transaction’s approval and transfer of the applicable assets. While network assets is the term used in the filing, I expect the deal includes Edgio’s delivery contracts and not their actual network assets tied to streaming and small and large object delivery. Lynrock won the assets for Uplynk and Interdigital, some of Edgio’s patents.

Adeia has sued Disney in Delaware federal court for allegedly violating its video streaming patents. Adeia also recently acquired some patents from Brightcove (for $6M), which originated from Brightcove’s predecessor, Unicorn Media. Some of the six Adeia patents in dispute involve “dynamic manifest generation, “key frame detection and synchronization,” and “user interface” methods.

Adeia says AT&T, Verizon, Cox Communications, Comcast, DirecTV, DISH, Spectrum, Frontier, Google, Paramount, Sony, TCL, Roku, and Samsung have all licensed Adeia’s media patent portfolio. Adeia was the intellectual property licensing unit of TiVo owner Xperi before Xperi was spun off into an independent company in 2022.

The patents in dispute are 9,762,639; 8,280,987; 9,860,595; 10,165,324; 8,542,705; 9,235,428. You can read the 168-page filing here: https://lnkd.in/eiqGgznC

Fed interest rate cuts and raises directly impact your job and your employer. It is important to understand the correlation it has on companies being able to grow and invest in more employees versus having to cut back and do layoffs. During the pandemic, interest rates on the money loaned by financial institutions were so low, in some cases 1%, that businesses called it “free money.” In March 2020, the Fed cut interest rates to zero and held them steady for two years. With such low interest rates, companies could borrow capital and expand their workforce, invest in R&D, launch new products and services and push into new territories.

However, once the Fed started raising the interest rate, the short-term rate banks used to borrow from each other, many companies could no longer afford to take on capital and had to do more with less, leading to layoffs. Below is an example of the rate companies are paying who took on funding, and as you can see, the interest numbers are no where near what they were during the pandemic. Keep an eye on these interest rates going forward, they have a direct impact on your job and your company.

• In April 2024, AMC Networks completed a cash tender offer to purchase outstanding 4.75% Senior Notes due 2025 and issued $875 million in new Senior Secured Notes due January 15, 2029, with an aggregate principal amount of 10.25%.
• In January 2024, Fubo closed a privately negotiated exchange regarding some of its Convertible Senior Notes. The old notes were at a 3.25% interest rate, and the new notes are at 7.5% interest if paid in cash and 10% interest if paid in kind. This extended a meaningful portion of its debt maturities out to 2029 from 2026.
• In May 2024, TelevisaUnivision announced that its wholly-owned subsidiary, Univision Communications, priced its refinancing of $1 billion of debt. This includes its offering of $500 million aggregate principal amount of 8.5% senior secured notes due 2031.
• In May 2024, Charter Communications priced $3 billion in aggregate principal amount of notes consisting of $1.5 billion in aggregate principal amount of Senior Secured Notes due 2029. The 2029 Notes will bear interest at a rate of 6.1%.
• In February 2023, Neptune BidCo US Inc., an affiliate of Nielsen, announced a private add-on offering of $500,000,000 aggregate principal amount of 9.29% Senior Secured Notes due 2029.

In the media, content, and cable industry, we have seen some interest rates as high as 16% on new loans and converting old debt to a low of about 5.5%. With the Fed having just announced its second rate cut this year, the hope is that interest rates will continue to go down, but it’s not guaranteed. All of this ties into the stability of your job and your company, and you need to watch the numbers and stay informed. Stay educated. The most valuable currency in the business world is (accurate) information.

Outside of the few content owners that operate their own CDN, third-party CDN vendors deliver nearly every byte of Internet-related information, content, and data that exists. For all intents and purposes, their customers are almost literally everyone in the world.

CDNs provide two essential components for how we consume media today. First, they enable the wide distribution of content using a 1:1 distribution method. Thanks to the Internet and CDNs, content providers can deliver content whenever and wherever it’s requested and get much more information about their viewers (and their preferences) than broadcast TV ever allowed. This means content providers can essentially understand their audience down to a single viewer with the goal of offering individualized experiences so they stay engaged and subscribed.

Second, CDNs provide the scale needed to deliver all this streamed content, which amounts to tens of billions of hours yearly. To handle this level of distribution, CDNs work with Telcos and ISPs to serve streaming media to everyone requesting it, wherever they’re located, in the best possible response time and with the best possible quality. That’s a staggeringly huge lift, and it’s only getting harder as VOD and live streaming grows.

CDN Assumptions: What Content Providers Should Know

A few common assumptions about CDNs have caused the bigger picture to be often misunderstood, and I’m determined to shed more light on these areas. These assumptions might seem small or insignificant, but with increased streaming demand and major shifts like live streaming sports, they could negatively impact the media industry in the long term.

• • Assumption 1: Large CDNs are connected to all ISPs, even remote ones
    While large CDNs are connected to remote ISPs in “theory”, the idea that they are directly connected can be false. Think of the CDN<>ISP ecosystem like an onion. Global CDNs are only directly connected to the handful of larger networks surrounding them. In reality, most ISPs are indirectly connected to large CDNs, sometimes multiple network hops away (in the outer onion “layers”). These ISPs are often smaller and serve less populated subscriber networks – a fact that comes with its own set of problems around upstream capacity and potential impacts on QoE, especially when network traffic spikes.

• • Assumption 2: Large CDNs ensure packets are always delivered quickly and reliably
    While global routes allow content providers to send packets anywhere, they have no control over the efficiency of packet delivery from the CDN to the next connected network – or even the pathways data packets take as they travel across the Internet. Content providers may know that the packet arrived, but how well did it flow to the ISP network further downstream? Did it have to be sent three times because it got dropped due to network congestion?From the somewhat centralized position of a large CDN, it’s difficult to validate and verify every connection to every ISP, especially those located further downstream. Unfortunately, content providers often have little visibility into how quickly or reliably packets reach their destination. Blind spots in content delivery and performance metrics are becoming a big problem that could impact revenue or induce subscriber churn.

• • Assumption 3: Recent funding (e.g., fiber to the home) means we can now deliver higher-quality service in hard-to-reach networks 
    Global initiatives like Project Gigabit in the UK have recently been greenlit to fund new ISPs and AltNets serving broadband and wireless Internet in remote networks. While this is a big step in the right direction, the reality is that managing content delivery just got more complicated. With further-reaching access to fiber, content providers may not realize the backhaul capacity issues that ISPs now face, especially with the surge in live streaming. When network congestion drives consumption upstream (like live sports events often do), everything is ultimately connected to the same large CDN, and every smaller ISP suffers.Some ISPs even have to rebuild portions of their network to handle the scale, which can be costly and disruptive. Scale problems are already complicated, and live streaming with large concurrent viewers is setting up an even bigger battle. Receiving more volume at a higher rate means less time to solve an issue. With subscribers quick to complain on social media and treating contract-free subscriptions as disposable, content providers should carefully consider whether they have the right delivery partners in place.

As the demand for video content and live streaming continues to multiply and consumption becomes ever more individualized, these issues will only get more amplified and complex. In upcoming posts, I’ll expand on these challenges and how content providers can get ahead of them. Here’s a hint: thinking smaller and getting more granular will ultimately support the bigger picture.

Qwilt and Comcast announced a deal that will see Comcast deploy Qwilt’s open caching platform inside its network at hundreds of locations over the rollout, with the first locations going live in the next 1-2 months. The companies are not disclosing the capacity Comcast will deploy within their network or the percentage of total video traffic that Qwilt’s Open Edge platform will deliver across Comcast’s network. However, with the addition of Comcast to Qwilt’s federated CDN strategy, Qwilt says their total on-net coverage of US broadband households will be 55% at the end of this year. While Qwilt has full US regional coverage like other commercial CDNs, its on-net coverage, with many embedded edge delivery nodes, distinguishes its operation and business model.

While Qwilt started as a point solution selling to ISPs, the company has transitioned to becoming its own federated CDN, selling delivery across multiple ISPs directly to content owners. It is starting to compete more with traditional CDN providers like Akamai, Fastly, and Edgio, using its federated CDN model to get a percentage of traffic from customers who use a multi-CDN approach. I have previously highlighted how I have seen Qwilt in the mix for a small percentage of live event traffic from Prime Video, Peacock and other large content distributors. With the Comcast deal giving Qwilt a much larger footprint, the company hopes to get a larger share of customer traffic going forward.

In the federated CDN model, Qwilt shares revenue with ISPs, as they will do in their deal with Comcast. A big advantage that Qwilt has over traditional CDNs is that the ISP provides the hardware—allowing Qwilt to add capacity at a fraction of the cost of competitors since Qwilt does not have to spend capex on edge infrastructure since the service provider owns and operates that part of the network. Comcast claims that by deploying Qwilt within their network, Comcast will create “the most distributed content delivery network (CDN) in the US.” While their wording is too generic, Comcast means most distributed within an ISP environment. Comcast will not have the capacity that Akamai or select other CDNs have in the US. Being widely distributed is crucial to offering a CDN service, but so is capacity, which must go hand-in-hand.

Meta and Deutsche Telekom are having a peering dispute, with the companies ending their interconnect deal. While the two sides are blaming each other, Meta is under no legal obligation to pay Deutsche Telekom for access to their network. Deutsche Telekom is under no obligation to give Meta peering for free. As Deutsche Telekom correctly points out, “Meta alone decides how Meta traffic is routed to Deutsche Telekom’s network.” Meta has switched to using transit providers instead of a direct interconnect to connect to Deutsche Telekom’s network.

In its blog post, Meta says, “given the court ruling concerning the unprecedented and unacceptable fees demanded, we are now routing our network traffic through a third-party transit provider.” Meta doesn’t say how Deutsche Telekom’s fees compare to Meta now paying transit providers to connect to Deutsche Telekom’s network. Deutsche Telekom disclosed that Meta used to pay for access to their network but stopped paying in March of 2021. Why did Meta stop paying? Was there a capacity issue? SLA issue? It’s rare for a platform provider to pay for a direct interconnect to a network and then just stop paying. Is Meta comparing Deutsche Telekom’s fees to what they had for free? Any fee seems “unacceptable” compared to what a company got for three years at no cost.

Meta says Deutsche Telekom “is using its market power to put its subscribers in Germany behind a de facto paywall, potentially restricting their access to internet services that do not pay Deutsche Telekom.” This is not accurate. Deutsche Telekom has not restricted or blocked any of Meta’s apps or services, and Meta knows this, which is why they used the word “potentially.” There is no business reason for Deutsche Telekom to restrict any of Meta’s services, as that would only harm Deutsche Telekom’s customers, who would complain to Deutsche Telekom, not Meta.

I see some in the media saying instead of Meta, “continuing to pay extortionary network fees for a direct connection,” which isn’t accurate. Meta hasn’t been paying anything for the past 3+ years. And what are the fees exactly? Who’s to determine what is considered “extortionary.” I also see some suggestions that Instagram, Facebook, and WhatsApp will “slow to a crawl,” which again is inaccurate. As Deutsche Telekom stated, “the rerouting of data traffic in the night from Tuesday to Wednesday went smoothly.” We see no reports of any Meta apps or platforms having QoS issues, and the media needs to stop suggesting that there are user issues when they do not exist.

While this all boils down to a business dispute between two companies, it’s important to point out that a German court ruled in this contractual dispute between Deutsche Telekom and Meta based on the principles of German contract law. Meta may not like the outcome of the ruling, but it’s the court that decided. Meta is free not to pay Deutsche Telekom and instead move to a transit provider model as they have done.

All of this aside, settlement-free peering and paid interconnects are the basics of how the Internet works so well. I disagree entirely with Deutsche Telekom’s statement that “regulations are needed to ensure the settlement of disputes.” Regulators have no idea how any of this works, and any time they do get involved, it is always behind closed doors. It becomes grandstanding by politicians and lobbying organizations paid by both sides, getting in the middle of the debate. Many step forth with studies and papers implying they are neutral when they get paid by lobbying organizations and have conflicts of interest. The public doesn’t get the details of the contracts or costs, and any shared data is always used to skew the argument without the complete data set being shown.

Regulators won’t fix this, nor should they be allowed to. The biggest problem in this whole discussion is the media. Too many suggest blocking, throttling or slowing down of Meta’s services is taking place when it isn’t. Or speculate, with no proof, that Deutsche Telekom won’t put enough capacity in place with Meta’s transit provider, which again, we know isn’t a problem. Analysts and media members need to stop taking sides and report the facts. All the what-ifs, what could, and what might happen are speculations on their part, and you can’t have a rational discussion and solve business problems focused on theory. Stick to the facts.

Deutsche Telekom Post: Meta is not above the law
https://www.telekom.com/en/company/details/meta-is-not-above-the-law-1079704

Meta post: Why We’re Having to End Our Direct Peering Relationship With Deutsche Telekom
https://about.fb.com/news/2024/09/why-were-having-to-end-our-direct-peering-relationship-with-deutsche-telekom/