Some Tech Details from Netflix’s Boxing Stream and What To Look For on Xmas Day
Since Netflix’s live boxing event, I’ve talked to more than half a dozen ISPs globally and others who had direct insight into traffic during the event. Based on those conversations, info from manifest files, error codes, and other sources I will not disclose, here’s part of what happened during the event and what to watch for during Netflix’s NFL games on Christmas.
Post-event, some quickly suggested that ISPs ran out of “capacity” while not even defining what capacity they were referring to. For most ISPs, almost all of Netflix’s traffic was carried on private network interconnects and appliances. Of the seven ISPs I spoke with, located in the US, Europe and another location, none had any capacity issues with peering, transit or private interconnects. In conversations with ISPs and what some ISPs posted online, they said:
- “We have plenty of capacity with them in the form of OCAs and Private interconnects.”
- “We had plenty of OCA bandwidth capacity and overhead available.”
- “None of our paths that feed our OCAs were at capacity.”
- “We had a healthy overhead of Netflix interconnect/OCA.”
- “None of our connections with Netflix were saturated.”
Multiple ISPs told me how they had the fight and regular programming streaming from an OCA closest to them, and only the fight had issues, which is very telling. That’s why Netflix’s SVOD content and other non-Netflix streaming services looked fine. Netlfix deploys caches differently based on the ISP and region of the world, with some ISPs having Netflix caches in some of their larger telco central offices and cable headends and others having direct private peering connections with them at core sites, among other connection options.
Netflix did not use any third-party CDNs for the boxing event and is not currently adding any third-party CDNs into the mix for the NFL games. Anyone who said that Netflix used them for the boxing event or wondered if they did should stop guessing. It’s public info, as you can see in traceroutes and by talking to those involved in the event. Netflix is utilizing the same infrastructure to deliver live streams and VOD content, and some statements by those on LinkedIn that Netflix is using a “separate” infrastructure for live is not accurate. Netflix spins up and down resources needed for their Open Connect servers and doesn’t have any hardware “limitations” across Open Connect for live like some want to suggest.
Looking at data from manifest files and other sources, from what I and others observed, it was interesting to see Netflix doing 1-second segments, which was surprising. One of the key technical issues I saw was that manifest files were not refreshing at times, leading to connection errors. I sorted through more than 1,000 posts on Twitter from users who provided screenshots of errors, and the three errors below were the most commonly reported from what I saw.
- Netfix Error TVQ-PB-101 (5.3): This error happens when an issue with data stored on the user’s device stops Netflix from playing. It requires a restart of the device.
- Netflix Error TVQ-PB-101 (3.2.400): This error happens when a network issue stops the device from reaching Netflix. Users who get the error on any device other than a Roku or smart TV are asked on Netflix’s website to contact Netflix to help them “investigate” the issue, and Netflix’s support site provides no suggestions for solving the problem.
- Netflix Error: UI-113 and NSEZ-403: This error happens when a network issue stops the user’s device from reaching Netflix and can’t connect to the user’s account, with Netflix suggesting to “try again later.”
Some LinkedIn posts suggested that Netflix should have prepared better for the event by “simulating” more traffic in advance, which is a ridiculous statement. Those making such comments show their lack of understanding of how live events work. No company can simulate 65 million simultaneous streams across any network in a real-world test. As Netflix said post-event, “It’s impossible for our engineering team to test that magnitude of traffic and viewership unless they have a real, live thing, which is what happened.” It’s also important to note that Netflix had no previous streaming telemetry and viewing data specific to live, in any significant quantity, to know from a regional standpoint where the most viewership would come from. They were using VOD viewership data to try and figure out live traffic, and it’s easy to warm up caches for VOD, something you can’t do with a live stream.
To tune stream performance during a live event, you need enough real-time network telemetry and observability data to make the tuning decisions. The boxing event was Netflix’s first large-scale live streaming event on their platform. I don’t know how many logs Netflix was processing per second, but for a recent live event with an AMA of just over 10 million, the content owner told me they were processing 15 million log records per second. With Netflix having so much data from the boxing event to put towards tuning QoE for the NFL games, I would expect the user experience for the NFL stream on Christmas to be vastly improved over the boxing event.
While no one knows how many simultaneous viewers Netflix will get for the NFL games on Christmas, I expect it to be less than the boxing event. The boxing stream had people tuning in from many different regions globally because the event was more of an entertainment spectacle than a pro boxing fight. The NFL games will see more traffic coming from the US, and Netflix knows what US regions will have the most viewership for the game based on the team’s local markets. That said, Beyonce performing at halftime is an entertainment event, not a sports one, so I would expect Netflix to see a surge of non-football viewers if they can time the halftime show accurately.
A not-often-mentioned but critical point is that Netflix’s boxing event was initially scheduled for July 20th. If the event had occurred on that date and not been postponed to November 15th due to Tyson’s injury, Netflix would have had 5 months between the boxing event and the NFL games on Christmas to help tune its network. However, due to the schedule change, Netflix’s 5-month window to prepare for Christmas dropped to 45 days. Those who do live events for a living understand the level of what Netflix is looking to pull off at a scale not seen before, with a timeline that is unheard of. Adding to the complexity is that a Netflix executive was recently quoted as saying that Christmas is the largest viewing day on their platform.
Using the simultaneous streams methodology, I estimate viewership will be in the 40-50 million range for the NFL stream on Christmas. The wild card in the guess is that Netflix has never given global streaming rights to any platform for an NFL game. In 2023, the Las Vegas Raiders and Kansas City Chiefs drew an average TV audience of more than 29 million viewers on CBS, making it the most-watched Christmas Day game in 34 years. If you compare that number to TV broadcasts from previous years, note that out-of-home viewing was not included in Nielsen final nationals until 2020. Between global rights, large viewership on Christmas day and Beyonce as the halftime show, there are elements to the stream where no previous data exists to make a comparison.
After the boxing event, there were more than a few posts on LinkedIn that gave out false information, made-up numbers, and opinions disguised as facts, using terms with no definition. Before you share any of these posts, please stop and look at the source of the info, any data or numbers presented, and the words used. These posts and their sharing don’t help the industry; they set us back. If we talk about the challenges of delivering video at scale with excellent QoE, the conversations need to be based on facts, not guesses or pitches for P2P products from vendors. Netflix publishes a lot of technical information about Open Connect publicly and presents at multiple shows that detail a lot, but not all, of what they have built.
My post does not capture all of the technical elements and challenges Netflix had during its boxing event. Still, quite a few pieces of information were shared on public platforms and in private conversations, providing a little insight into what took place. Some additional details I have can’t be disclosed but have been used on background. For Netflix’s games on Christmas, I have put together a team of people from across the industry who are in the infrastructure trenches and will have access to real-time stream data. They will contribute information to me during the event and remain unnamed as I blog about it. If Netflix pulls off the NFL games with minimal hiccups, which is my hope, I’ll have nothing to blog about, which would be good. Their success in streaming the NFL games would be a success for the entire streaming industry, and we should all be rooting for them. In this case, the progress of one would be the progress of all.