Come by the Streaming Mixer Wednesday October 16th at the Streaming Summit in NYC. Network and have a drink with industry peers thanks to Limelight Networks, Conviva and Verizon Media, from 5-6pm on the show floor of the NAB Show New York. Contact me for a free exhibits pass to attend.
Sponsored by
I am please to announce that Amazon Web Services (AWS) is offering FREE hands-on workshops at the Streaming Summit, taking place on Oct. 16th in NYC, led by their Training & Certification/Media Solutions Architect teams. Learn “How to design & build well-architected, serverless media solutions”. You can register for free here and also see a list of all the classes. Learn how to build well-architected, serverless media solutions using AWS Elemental and Amazon Web Services API-driven artificial intelligence and machine learning services. Developers (including Asset Managers, Digital Team Members, DevOps and Streaming Engineers) are encouraged to attend.
Thursday, September 26th at 11am ET, Frost & Sullivan will be hosting a webinar entitled “Investing in Your Enterprise Video Strategy – Don’t Get Left Behind“. Hear the best practices on how to articulate the value of the investment to gain internal agreement and how enterprise video can deliver real business results and complement the digital transformation agenda. Learn about:
Register for free and hear from two Frost & Sullivan analysts on this important topic.
Last week, CenturyLink announced they had acquired privately held Streamroot, a CDN provider with an underlying P2P technology. Streamroot had raised $6M to date and the company is expected to do sub-$5M in revenue for this year. Based on what Streamroot’s valuation would have been for a Series A round, CenturyLink valued the company between $20M-$30M.
CenturyLink’s acquisition is motivated by improving the performance for viewers of live events and day/date content releases to reduce buffering and failed starts during peak hours and catch up TV. Streamroot leverages a software-based mesh network and a deterministic data science methodology, providing device awareness to ensure that consumers have only optimal experiences without creating any data privacy concerns. In short, this extends the capabilities of CenturyLink’s existing CDN network to offer the promise of better performance in hard-to-reach places and during volume spikes when delivery is most difficult. Of course, we have heard these value propositions before, however, the vision that CenturyLink and Streamroot share and what’s under the hood is admittedly more nuanced.
While most previous P2P attempts have failed at scale, Streamroot has actually delivered premium content with million+ simultaneous viewers with national broadcasters like TF1, Canal Plus, and RTVE in Europe and LATAM during the World Cup, Copa América, and others. To date, most vendors that have focused on P2P have tended to roll out “cheap” CDN alternatives, but Streamroot has been quick to identify that “cheap” wasn’t what the market wanted. They had to deliver visibility to material QoS improvements while handling complex workflows including proprietary ABR, multiple SSAI providers and DRM considerations.
CenturyLink said what makes them combination of the companies stand apart, is in their belief in using data from the device to improve content delivery at both a micro and a macro level. At a micro level, in the ability of Streamroot products to take into account each device’s instantaneous conditions to select the best delivery source; on the macro level, in the data gathered on the network topology of ISPs globally, of device behavior depending on OS, version, content type, etc. In other words, this isn’t just “peer-to-peer.” It’s about device telemetry; it’s about adapting delivery to each individual user and content and network; it’s about broadcasters being able to customize a technology solution to their unique use case and all the variables that come along with it: encoding ladder, ABR, ISP peering, networks, topology, and more.
Thinking holistically about delivery has allowed Streamroot and CenturyLink to find a common avenue to improve content delivery. And this offers applications far beyond just P2P for video. Client-side load balancing, file download, ISP-aware routing and devices localizing to the best possible topological and performance based cloud edge serving locations, could all be use cases. They have the potential of using device-side technology to provide the best user experience, helping ISPs route traffic flows more efficiently and helping content providers, web applications and edge cloud services localize connections.
Client-side integration is never a pleasant subject but we might as well take it as a given in a multi-CDN, multi-OS, multi-hardware world. With Streamroot, the company says pre-integrations on pretty much every major web player make web deployment possible in a matter of minutes. Streamroot has also worked to make its mobile SDKs light years simpler than the average advertising – whether SSAI or CSAI – or DRM deployment. It essentially boils down to roadmap prioritization, as the actual man-hours to integrate the solution on any device can be counted on two hands. And when benchmarking that again the quality improvement, the company says the ROI calculation is pretty simple.
I’m very interested in seeing where CenturyLink is going with this acquisition and with its media delivery services in the short term. The CenturyLink team is making smart moves to incorporate technologies that may very well redefine content localization in an otherwise lackluster market, especially in Europe. Right now the Streamroot technology works just for video content, both live and on-demand, but CenturyLink said they plan to invest in the technology, add more developers, and add download functionality in the future. It’s a good move by their part to pick up some proven technology and a good engineering team in Streamroot, and overlay it on top of their network, supported by CenturyLink’s sales and marketing team.
Reducing video rebuffering can be difficult. One solution that many people are talking about these days is moving to a multi-CDN architecture, a topic I’ve written a lot about. But will going multi-CDN magically reduce your rebuffering and drive away all of your streaming ills? The answer, of course, is complicated. Going multi-CDN can provide several benefits for customers, such as better geographic coverage and, possibly, better economics. Adding live switching logic between the CDNs goes a step further and enables load balancing and redundancy in case of problems. But what are the problems that customers are likely to encounter? Let’s examine some common CDN problems and their impact.
Catastrophe and Chaos
Once in a blue moon, a CDN will experience a major outage that affects a large geographic area. These outages are so extreme that they shut down a large portion of the Internet for a non-trivial amount of time. Recent examples of such outages:
Detectability: Very easy to detect as any metric that you care to measure will explode. Your alerts will fire or, if you don’t have alerts, you’ll get messages and phone calls from users.
Solution: A good CDN switching engine will attempt to re-route users to a working CDN within the limits of the outage.
PoP Flop
Occasionally, a CDN might experience a local issue in one of its PoPs (Point-of-Presence), which means all of the users that are routed through that specific PoP will have problems fetching video segments, and will most likely experience rebuffering.
Detectability: Medium or hard depending on the percentage of users that are affected. A large portion of the traffic would skew the metrics enough to create an anomaly, while a small portion might get swallowed within the geographic granularity of the monitoring system.
Solution: A good CDN will re-route the impacted traffic to a different PoP within its network which will cover for the faulty PoP with best-effort performance. A good CDN switching engine will eliminate the faulty CDN altogether from that region and just use a non-faulty one.
Smaller scale outages are so hard to detect that you might never know they ever happened. Have your users experienced such an outage? The answer is most likely yes, since all CDNs see dozens of them as part of their daily monitoring efforts. In the industry, we call these events “blind spots”.
The Blind Spots of CDN Switching
There are 3 main blind spots that server-side CDN switching engines do not address very well:
Unfortunately, not all regions have fresh performance data all the time and so a fallback logic is usually applied. When there isn’t enough data in a specific region, data from its greater containing region will be used instead. It’s possible that a region with small amount of users gets swallowed up by a larger fallback region, in which case an outage might not be detected at all because the affected users comprise a small portion of total traffic that is not enough to “move the needle”.
This is a data granularity problem. A broadcaster might have 100k users that are spread across 15 countries, 1,000 unique regions, 5,000 ISPs and a host of other parameters. Taken together, these parameters segment the user base into millions of tiny dimensions, none of which will have enough data to perform meaningful switching decisions, not to mention the load it will create on the switching system. For this reason, server-side switching is inherently limited to a more coarse grouping that is technically and mathematically viable. This reality creates a blind spot when it comes to smaller regions that might get hit by a local, undetectable outage.
Real World Example of The Granularity Problem
In the last week of August, an outage occurred in the U.S. which demonstrates the granularity problem. An outage by a CDN I won’t name, caused a significant drop in request performance that, in turn, led to rebuffering. Thanks to Peer5 for sharing the screen grabs (below) from their monitoring tool showing that at 7:20 AM, an increase in Time-To-First-Byte(TTFB) was observed from an average of 850ms to a peak of 6700ms. When comparing the 95th percentile TTFB of the affected area to its greater containing region, it’s clear that the affected area didn’t constitute enough data to move the overall metric.
95th Percentile TTFB – Affected area vs Greater Region
The greater containing region (blue) doesn’t show any anomalies throughout the outage. (less is better)
Rebuffer Time as %
Rebuffering spikes render the playback unwatchable. (less is better)
Allocated CDNs Over time
While this chart might seem dull, it illustrates the fact that throughout the entirety of the outage, no CDN switching took place for the given region.
Enter: Per User CDN Switching
Video playback is a very fragile thing. A user might have just a couple of seconds of content buffered ahead and any slowdown in fetching segments can easily consume that buffer and freeze the playback. For this reason, vendors in the market are coming up with ways to fix the problem. For instance, Peer5 created a client-side switching feature which constantly monitors the playback experience for each individual user and is able to react to poorly performing CDNs within a split second (literally, milliseconds) and prevent rebuffering from ever happening. This means that even an outage that only affects one user will be accounted for. The below charts shows the performance during the outage described above with and without a client-side switching feature.
95th Percentile TTFB – Affected area vs Greater Region
The TTFB of the client-side switching group (green) was affected as well but much less than the other group. (less is better)
Rebuffer Time as %
The client-side switching group (green) experiences almost no interruption in playback. (less is better)
As seen in the graph above, users that relied solely on server-side switching (red line) were impacted significantly, compared to users with client side switching. Server-side CDN switching was not granular enough to detect the local outage and the assigned CDN for that region remained the same even though some users experienced terrible performance degradation. The client-side switching, with its per-user granularity, was able to change the mixture of CDNs within the region and avoid the issue in real-time. The rebuffering was reduced from 11.2% to 0.2% for client-side switching enabled users, and the overall region rebuffering was reduced by 70% from 1% to 0.3%.
Summary
When CDNs experience outages, users will encounter rebuffering. There are multiple types of outages, some will go below the radar completely undetected while some will make you notice them immediately. Different layers of redundancy and different levels of granularity tries to address the various outages an online delivery pipeline might experience. A combination of several such redundancy tools is likely to achieve the best UX. Employing server-side switching alongside client-side switching allows customers to:
There’s lots of ways to solve the video buffering problem depending on what type of video you are delivering, (live vs VOD), the platform or devices you are delivering it to and the user-experience you are looking to achieve. What’s your take on the best ways to reduce video buffering? Feel free to leave them in the comments section.
I’m pleased to announce the first set of speakers for my Streaming Summit, at NAB Show New York, taking place Oct 16-17. The program schedule has also been added to the website and when completed, we’ll have over 100 speakers, across two days of the show. Newly added speakers include executives from CBS Interactive, Quibi, NBC Sports, YouTube, Twitter, Amazon Fire TV, WarnerMedia/HBO – with lots more on the way! Register before Sept 12th using code “early” for a discount on your ticket. You can see the entire agenda on the schedule page. #streamingsummit #nabshowny
Thanks to Beamr, Mark Donnigan and Dror Gill for having me on their “Video Insiders” podcast to talk about on Disney+, Quibi, HBO Max, Hulu, ViacomCBS, and what the forthcoming D2C launches mean for incumbents, including Netflix and Pay TV operators. Hear my thoughts on content aggregation and ideas for measuring success in OTT, along with the technical plans and platform choices being made by these developing services. This is a frank and honest, real-time and real-world conversation about the groundswell of direct to consumer OTT services which will be unleashed over the next few quarters.
You can listen to the podcast here: https://thevideoinsiders.simplecast.com/episodes/episode24