Archives

[Updated 3:27pm] Amazon has put out a release saying they averaged 15.3 million viewers across all platforms for their first exclusive TNF game. Their number, which is different from Nielsen’s, includes set-top boxes, connected TVs, web and mobile, as well as Twitch, local over-the-air stations, out-of-home viewing, and NFL+.

[Updated 11:33am] The 13 million figure includes local over-the-air simulcasts in the home markets. Excluding those simulcasts — around 602,000 viewers on LA FOX affiliate KTTV and 555,000 on KC NBC affiliate KSHB — Amazon Prime alone averaged in the neighborhood of 11.8 million.

[Updated 11:12am] CNBC is reporting that “according to Nielsen”, Amazon averaged 13 million viewers based on data released by the company. As of now, there is no press release on the wire or on Nielsen’s website yet.

[Updated 10:14am] Shortly after posting, I see one article is reporting the number is 13 million viewers, with the source being Nielsen, but I don’t see any official release from Nielsen as of yet. I see a second article saying the number was over 15 million viewers, also attributed to Nielsen, so lots of conflicting numbers. I’ll update this post if/when Nielsen puts out a release.

———————————————

Multiple advertisers tell me Amazon had between 10 million and 11 million total “viewers” for their NFL Thursday Night Football game on September 15th, with the actual number being closer to 10.5 million. A week after the game, Amazon and Nielsen still haven’t put out any numbers, which is odd. Leading up to the game, the media reported that Amazon was telling advertisers they were targeting 12.5 million viewers. If both of the numbers being reported are accurate, Amazon fell a little short of their goal. Amazon has been quoted as saying that their “audience numbers exceeded all of our expectations for viewership,” but we don’t know what their expectations were. If they were 12.5 million viewers, then the 10 million – 11 million number sounds low. [Updated: The numbers are low as Nielsen says the number is 13 million “average” viewers.]

Another key point is that the term viewer is vague, so without knowing how Nielsen and Amazon define what a viewer is, there’s no way to know what the true reach of the audience was. If a user streamed the game at kickoff time and then left, only to come back later to the game, were they counted as a single viewer or two viewers? Also, of the total number of viewers that were counted, how many were unique? And how is Nielsen counting multiple people in the room watching from a single device on a large screen?

We also don’t know what type of engagement Amazon saw during the game since the live stream was embedded on the home page of Amazon.com. If a user was on the website to search or buy something, but never made the video window larger, or clicked within the video window at all, were they considered a viewer? And, to be counted as a viewer, did the video stream have to play for a minimum amount of time? I don’t expect Amazon and Nielsen to give out many details, but knowing the number of unique viewers, average viewing time per user, number of simultaneous viewers (not just Average Minute Audience) and breakdown of viewers on large versus small screens is needed to judge the success.

On the tech side, from what I saw personally and was reported to me by other viewers, the lowest bitrate looked to be about 5.5Mbps and the highest around 7.5Mbps. So that would put the average bitrate delivered at around 6Mbps. For those like myself that were experiencing delivery problems, I didn’t see any issue with the multiple CDNs I saw Amazon using. All issues seemed to be with some ISPs and in particular, Comcast. I don’t know what the root cause of the ISP issue(s) were, but for all the talk within the industry of live events being “limited” by CDN capacity, the problem on QoE was once again down at the last mile level, not the CDNs.

The media has reported that based on an internal email Amazon sent out that Amazon, “saw the biggest three hours for U.S. Prime sign ups ever in the history of Amazon.” We don’t know exactly how many signups Amazon got and the key metric to watch is how many stay on after the 30-day free trial ends. For some NFL fans like myself that only watch one team, there is no reason to pay for a Prime account past the 30-day free trial if your favorite team doesn’t play more than once on TNF, or at all. For instance, the Giants are never on the TNF schedule and the Jets only play once on Thursday.

During the game I saw people on Twitter talking about how they shared their login with others so it would be interesting to know the total number of Prime accounts that were used to stream the game. It won’t be a 1:1 ratio of accounts to viewers, so the number of Prime accounts used would have to be less than 10 million, if the 10-11 million viewership number is right.

It’s interesting to see how Amazon isn’t acknowledging any of the technical issues in how they talk about the event. In their email to employees, Amazon references the, “stellar production of the game, to the quality of the stream customers watched at home.” We don’t know the exact number of viewers that had problems with the video and audio, but it was not isolated. Amazon’s President and CEO pushed out a Tweet during the game and was quickly met with 194 comments from viewers complaining about the quality of the video and/or audio. Across social, there was thousands of negative comments on the quality of the stream with viewers experiencing technical issues. Amazon should just address the issues some viewers had and talk about how they plan to improve upon them for the next game.

I’m sure some will say this was Amazon’s first TNF game this year so it is expected to get better over time, which very well could happen. But they are missing the point. Unlike other sports like MLB that have been streaming for 20 years, historically the NFL has always been a sport that most viewers have watched via cable or satellite. It works every time, with nearly the exact same quality no matter the city you are in. The cable company controls the set-top-box you are getting the video from and the pipe it’s coming across. The experience a viewer has is not based on their technical setup and it always works. It’s easy to use.

Streaming is the exact opposite and “maybe” you can have a good experience, or maybe not. In some cases, Amazon might control the hardware device, but they don’t control the last mile and can’t force ISPs to do what they want. Amazon can produce the best quality video possible, but once it hits the ISP, the success or failure of the stream is now in another company’s hands. Streaming can’t replace live cable TV as a distribution medium with the same reliability, scalability and stability. This is the same with any streaming service not being delivered by a cable operator and as a result, for the first time, the NFL is making fans wonder as they go to watch their favorite team “if” the video will even work. That’s not something NFL fans have had to wonder about in the past – until now.

Podcast Episode 34 is live! This week we breakdown Amazon’s stream of their NFL Thursday Night Football game on September 15th and the wide variety of user experiences that were reported, both good and bad. We discuss the issue that some ISPs had resulting in poor video quality, the terrible job by Amazon support, a large group of people reporting audio and video issues and a host of other comments from viewers. We also highlight how the Amazon game is an example of why streaming will not replace cable TV as a broadcast medium with the same reliability, scalability and stability. Thanks to this week’s podcast sponsor, Agora.

Companies, and services mentioned: Amazon Prime Video, NFL, MLB.

[See below for updates] Starting at 7pm ET, I’ll be live blogging Amazon’s Thursday Night Football video stream across many devices and watching comments from users with their feedback via social and chat rooms. I also have users from around the US dumping me their Charles logs so I can see the quality others are getting. (Contact me if you want to help provide feedback) I’ll provide as many technical details on the stream I can throughout the night. I’ll break out where I see any technical issues versus people complaining they can’t see it due to user generated issues or confusion on the service.

11:13pm ET: Thats’s a wrap. Overall, terrible job by Amazon Prime Video with the audio and video quality in particular, but a host of other issues as well. Their support clearly isn’t ready to field questions from viewers and the website has conflicting support details. Some ISPs had some major issues resulting in poor video quality and long startup times. Numbers will be out soon from Amazon and Nielsen, but don’t expect anything more than the AMA (Average Minute Audience) which won’t tell us how many users who were watching had terrible quality. Don’t expect any data to come out on max bitrate, startup times or what percentage of users had rebuffing. Also, how many refunds did Amazon give out? Doubt they will say.

11:00pm ET: All users are supposed to be able join the live broadcast, pause, and rewind up to 15 minutes from the point that they turned the game on, but am seeing some comment that they can’t pause or rewind at all. To watch at any point in the game, users would of had to have turned on the DVR feature before the game started.

10:44pm ET: Amazon has been offering some users a 1-month refund for issues they are having with video quality. They just offered me one, (I declined) but that’s not a good sign for them from a P&L standpoint. They paid a lot for these games.

10:12pm ET: Some users reporting Amazon support being unavailable with a message of, “We’re having trouble receiving chats right now. We’re very sorry and we expect to have chat service fully functioning again soon. Please try again later or contact us by phone or email.” At one point, support told me to email them a screenshot to “cs-reply@amazon.com” but it bounced back saying “this address does not accept incoming direct e-mails”.

9:55pm ET: The number of users reporting video quality issues on Twitter has been constant all night with no slowdown. Amazon support told me they had a “server issue” but I don’t think the support person has any idea why there are problems. I do not believe this is a third-party CDN issue, it looks to be some last-mile ISP issues for at least two ISPs I’m aware of. Not sure if I will name them or not, let’s see how Amazon addresses the tech issues after the game. If they play down the issues and pretend it wasn’t a big deal, that would be very bad on their part as they weren’t “isolated” issues for just a few viewers. Also, it’s not as if this game was in 4K or they were trying to deliver high-bitrates.

9:11pm ET: Amazon Prime support isn’t trained or ready for this game telling me I need “15 Mbps for Ultra HD”, but the game isn’t in UHD (website says 25Mbps for UHD). I’ve dealt with six different support people now, asking me for my “purchase ID” and one telling me to do “ASTRO troubleshooting,” which I don’t know what that is and he could not explain it.

8:56pm ET: Some ISPs are clearly having some issues with QoE across their network which looks to be the root cause of some of the video quality issues. One ISP told me they are seeing “issues”, but I’m not going to name who they are.

8:45pm ET: I’ve been watching the PrimeVision feed but have yet to see any stats “overlaid” on top of the video. Unless that’s not how it’s suppose to work, but I only get stats around the video.

8:37pm ET: On my MacBook Pro (16-inch, 2021), it’s averaging 10 seconds for the stream to start.

8:28pm ET: I can’t keep up with the number of users reporting issues on Twitter with the video quality and/or audio sync issues, which are the primary complaints, also video frozen or video being pixelated. A complaint is coming in about every 10 seconds on Twitter right now.

8:19pm ET: Amazon is working with NBC Sports to produce the games, utilizing a lot of the backbone of NBC’s infrastructure at 30 Rock and Stamford for delivering the signal, commercial insertion, and overall master control.

8:08pm ET: I’m running the stream on just one Fire TV Max and it froze on me. Had to restart the device to get it stream again. Some can’t get the stream at all with an error of “Please try again later“.

8:02pm ET: Long list of users on Twitter complaining about the “picture quality”, “the picture is terrible”, “video is choppy as hell”, “picture is terrible”, “video quality is terrible”, “not clear at all”.

7:53pm ET: One of my Amazon Fire TV sticks is getting an “error code 9001” when I try to restart the stream. Amazon support chat was very quick to get someone, but they didn’t know the cause and suggested someone call me which they did. Problem is that Amazon tech support is asking me for an “order ID” for my purchase, which of course doesn’t exist. I’ve now been passed to a third support person who’s asking me for the “name of the football game”. Have now been passed along to a 4th person. 42 minutes into being on the phone, support doesn’t know what the error code means.

7:41pm ET: Amazon has posted a message on their website saying, “We’re sorry for the inconvenience, we’ve temporarily disabled several rows on the homepage of Prime Video. To access any show you want, please do so via search and navigate to the next episode you’re interested in.” I have seen a few users commenting that they cannot find anything relating to TNF on their Prime feed, so I’m speculating Amazon has made this change to make it easier to navigate to the game.

7:31pm ET: Amazon’s help page says you need “3MB/s for SD content, 5MB/s for HD content and 25MB/s for UHD.” But the TNF stream is not available in UHD, so that might confuse some users. Amazon has a users forum setup here for tech questions and issues.

7:27pm ET: Some users are getting a “error code 9001.” Unless you have enabled the record option previously, you will not be able to watch past Thursday Night Football events. But I don’t know what that error code means with regards to the live stream.

7:20pm ET: The audio feed and mix for the pre-game is a bit off, volume changes between the show and commercials and isn’t balanced right. I’m seeing comments from others of “TV is almost maxed out on volume and the commercials are blaring loud.” My TV volume is on 10 and it’s too loud.

7:15pm ET: A few comments on social media of users saying they need to “improve the video quality” and “upgrade the bandwidth and get this stream straightened out”, “the image quality sucks. Is it going to get better for the game?” and  “we want 4K video!!!” Also a comment of, “Quality is horrible for the game PrimeVideo. Looking at my router stats, the bitrate is the issue. Fix it!”

7:09pm ET: Some users on social media are reporting audio syncing issues. I am seeing slight un-sync issues at times, but for me, it’s very, very slight and almost unnoticeable.

7:03pm ET: The pre-game stream is live and running across all my devices. Looks good on smaller screens, but on 65″ and 70″ TVs, you can see it’s a low bitrate and slight stuttering at times. Audio is perfectly synced for me. I do see some latency across all devices with some devices 2-3 seconds behind others.

Testing Details: My running two different ISPs at home with a 300Mbps with Verizon FiOS and 300Mbps with Optimum. I’m testing on 7 different Fire TV Sticks/Cube (Fire TV Stick 4K Max, Fire TV Stick 4K, Fire TV Stick Lite, 2nd-Gen Fire TV Cube), MacBook Pro’s, iPad’s, iPhone and smart TVs (LG, Samsung, Vizio), with all having the latest OS and app versions installed. All Fire TV Sticks and smart TVs are connected via ethernet and tablets are on WiFi. iPhone testing is being done with both WiFi and 4G (Verizon).

Podcast Episode 33 is live! This week I’m joined by a special guest Barry Tishgart, who just left Disney Streaming Services as a SVP and GM of Technology Services. Barry joins me to discuss the current state of the sports streaming business including RSN licensing, bundling, pricing, distribution, video quality and what the consumer sports experience will look like in the future. Thanks to this week’s podcast sponsor, Agora.

Companies, and services mentioned: Amazon Prime Video, NFL, ESPN+, MLB, Nielsen, Peacock TV, NBC Sports, Bally Sports+, Sinclair Broadcast Group.

I’m pleased to announce the call for speakers is now open for the one-day NAB Streaming Summit, taking place on Tuesday October 18th as part of the NAB Show New York at the Javits Center. The summit will be a two-track event that will focus on the “Metrics, Measurement and Monetization” of streaming video.

Speakers from the OTT, sports, advertising, media and broadcast industries will discuss the technical and business challenges and opportunities about what metrics should be collected (subs, viewers, APRU, engagement, QoE etc); how measurements should be combined (player, ad networks, Nielsen, CDNs etc.); and how the data should be used to determine monetization models (churn, retention, AVOD vs. SVOD, P&L etc.)

Please see the website for all the details on speaking and sponsorships and please reach out to me with any questions.

•  I looove watching lagging frame by frame video on TNFonPrime. Unreal how bad this video quality is!
• Picture quality is just okay. Audio mix is just okay too. Side note: too many clicks to get to the game
• The picture is terrible on TNF ugh
• Can you up the bandwidth so we can see better than a 480i picture?
• Why is this stream of this game so choppy? None of my other apps do this on my firestick
• Can you fix the lag between the video and audio. Audio is way a head of the video.
• Only three minutes into game before the stream froze and said “something went wrong. Please try again later”.
• Already have problems with glitches and now no volume.
• Audio on my prime stream is about 3 seconds ahead of video. Hear the whistle before the play is visibly over.
• Already annoying with the fact theres 5 steps to just get to the game anytime I change back to normal TV from the app.
• The streaming quality for TNFonPrime is straight garbage. Can’t keep a stable resolution
• Anyone else’s TNFonPrime stream super laggy and freezing constantly?
• Is it just me or is the TNFonPrime stream super choppy and stutters?
• Running like trash on my 55” Roku TV. Awful resolution drops, freezing, stuttering, audio desync. Really no excuse for this when Prime Video has done plenty of live TV in the past.

To be clear, I have no way to know what percentage of viewers had problems like I did or what percentage had no problems. We also don’t know how large the audience was, the platforms and devices they watched the game on and how much viewing was on small screens versus TVs. This game was basically a live “rehearsal” for Amazon and I would expect them to improve upon the experience for their next game on September 15th. It’s possible they will make some changes to their bitrate ladders and other settings, but the audio being out of sync so badly and the bitrate seemingly looking too low are big issues that need to be addressed when the NFL season kicks off next month. Streaming is not a perfect technology and never will be. That’s not how the technology works. As a broadcast medium, it will never deliver the same experience to 100% of viewers. That’s just reality.

Notes: I tested on 4 different Fire TV Sticks/Cube (Fire TV Stick 4K Max, Fire TV Stick 4K, Fire TV Stick Lite, 2nd-Gen Fire TV Cube), MacBook, iPad, iPhone and smart TVs, with all having the latest OS and app versions installed. All devices were also restarted before testing and for some, during testing as well. All Fire TV Sticks and smart TVs were connected via ethernet, with tablets on WiFi and phone WiFi and 4G. My internet connection is 300Mbps with Verizon FiOS and 300Mbps with Optimum.

During Akamai’s Q2 earnings call, the company noted that revenue attributed to their delivery business was down 11% year-over-year. These numbers lead some to speculate that Akamai had lost a large percentage of market share, which isn’t the case, or that CDN pricing was falling faster than usual. While there has been some accelerated pricing pressure in the CDN space, but only for a handful of the largest CDN customers across the industry, the reality is that something bigger is going on.

Since the start of the year, many streaming services have focused on doing a better job of optimizing their bitrates and in some cases, reducing their bitrate ladders in an effort to save money. Given all the cost cutting everyone is doing in anticipation of recession like macro economics, many streaming services have cut back on the number of bits they are delivering. And it’s not just Akamai that is seeing the impact of this across their network. Other CDNs I have spoken to have also seen customers take this approach and some ISPs I have spoken to have seen a drastic reduction in the number of video bits delivered this year as a result of the bitrate optimization.

One ISP in the US that didn’t want to be named told me, “Since the beginning of the year our combined peak ingress internet traffic (IP Transit, Public and Private Peering, CDN and Caching) has been down more than has been in other years. It is not unusual to be down or flat till this time of the year, but even with subscriber growth we are down this year. I am thinking our year end traffic growth will be lower this year as we are still negative for the year.” Checking with a few other ISPs, they are seeing the same thing, an overall rate of fewer video bits delivered across their network this year.

Another key data point is that 4K/UHD still makes up a very small volume of the overall amount of video bits delivered across CDNs and ISPs and is not seeing much in the way of growth, year-over-year. Multiple CDNs have told me that of all the bits they will deliver this year, 4K/UHD will make up “less than” 10% of those bits, which is nearly identical to last year. For one OTT service I spoke with, they told me that less than 2% of total viewing time across their platform this year was in 4K/UHD content and they are one of the largest OTT platforms in the world. For all the talk of 4K in the streaming industry, we have yet to see much in the way of real adoption based on the total volume of hours viewed or total bits delivered.

When some OTT services first came to the market they pitched 4K/UHD to consumers as a way to differentiate their service. But to date, consumers have not seen 4K/UHD as the value proposition some expected. Some streaming services including fuboTV and YouTube TV charge more each month for 4K quality, as does Netflix. None of these services will break out for the industry what percentage of their users are paying extra each month for 4K/UHD quality because the rate of adoption is so small.

Part of the reason consumers are not demanding 4K/UHD content is because HD streams look great and the industry started applying HDR to HD streams instead of how it was originally rolled out, which was only with 4K. Today, HD streams with HDR look so good that many consumers would not be able to tell the difference between HD/HDR and 4K/UHD. And for those that do, they simply don’t see it as a big enough feature to pay for it. Also, when talking about the total volume of bits delivered, we have so much viewing done on desktop, tablets and mobile where 4K can’t even be used. We’ve also seen very few tentpole streaming events or pieces of content in 4K/UHD including sports. As of now, Amazon’s Thursday Night Football games won’t be in 4K/UHD and Apple’s Friday Night Baseball games aren’t in 4K/UHD either. Peacock has announced they plan to stream Premier League games in 4K, but not until 2023. Very few major sporting events being streamed are in 4K/UHD and we all know that sports is where a lot of viewing comes from.

We also see services like Netflix that set the default streaming quality to “Auto” in their settings, meaning the best possible mix of quality and data usage. I’ve never seen 4K on by “default”. While the recent premiere of House of the Dragon was available in 4K/UHD on HBO Max, you won’t see Warner Bros. Discovery talking about the percentage of viewers that watched it in that quality. Without going into things I can’t discuss, I can tell you the numbers were much lower than many in the streaming industry would expect.

Not all streaming and OTT services are being more aggressive in optimizing their encoding or removing higher-tier bitrate ladders. Looking at how Disney is doing encoding today for their D2C services, versus the start of the year, doesn’t result in any changes I have seen from looking at the streams on my devices. But in some cases, the Disney app limits the video streaming quality to 1080p depending on the platform. The PS5 version of the Disney app doesn’t allow for 4K streaming and caps playback at 1080p, although rumors of an updated Disney app for PS5, codenamed “Vader”, is apparently in the works.

From my point of view, the lack of any large-scale 4K/UHD adoption by consumers is not a bad thing. It doesn’t help third-party CDNs that are in the business of delivering video bits and have an economics of scale business, but any streaming tech should only be used when it is driven by a clear business purpose. For some, 4K/UHD matters and streaming services can charge those consumers more money. But for the vast majority of viewers, based on data from streaming services, CDNs and ISPs, 4K/UHD simply isn’t something consumers are demanding or willing to pay more for.