New Chart Shows The Differences Between Performance Monitoring Vendors

Performance monitoring was a very straightforward category of service just a few years ago. Companies set up programmatic test boxes in data centers around the world and pinged services designated for testing as their business model. Companies like Keynote and Gomez provided these synthetic monitoring services, and these companies were icons in the field, responsible for creating the market. However, there hasn’t been many innovations in their services over the years, and competition was primarily always based on price. Because the locations of these servers performing the synthetic monitoring could be discovered, these services were also gamed by smart CDN’s that wanted good scores. As a result, this lead to cynicism regarding the results, making their data much less effective.

The good news is when it comes to performance monitoring, things have evolved rapidly in the last few years. The main reasons for this are that application monitoring, which was traditionally data center centric, has now been brought into the cloud. New pricing models such as Cedexis with its free Radar product are creating a commoditization of the traditional service in favor of a more valuable service (in their case the global CDN load-balancing product Openmix). In addition, RUM (Real End User Measurements) has come to the forefront as the defacto standard for monitoring and is a far cry from the methodology used in the early days by systems like Keynote and Gomez.

These three factors have made a market that was feature stable and resistant to price decline into a completely new market that defies easy categorization. The chart below takes a look at some of the current performance monitoring competitors in the market and details their focus. [UPDATE 10/3: The first version of this chart incorrectly showed Keynote as not supporting RUM. Keynote does have RUM measurements taken by inserting a W3C approved tag on the site.]

As the chart shows, the competitive landscape is broad, and the features supported vary widely depending on the focus of the company. The prevalence of RUM measurements is also something to take note of, and while the majority of these vendors claim to have some form of RUM, it can be deceptive. The reality is that WHAT is being measured by the RUM also matters. Cedexis for instance measures every CDN (and Cloud) for the purpose of their load balancing solution. These measurements are of specific objects that have been inserted into the CDNs for this purpose. This gives a very clear view of the relative performance across CDNs. Other RUM measurements by some of the other vendors may be more directed at specific customer infrastructure, which could provide more relevant reporting only for a specific customer.

Cedexis is clearly the winner with their billions of RUM measurements a day and the fact that they have built a community out of some of the most ferocious competitors in recent history, namely the CDN vertical. This cannot be underrated as the community that Cedexis has built is not easily replicated and their “Wisdom of the Crowds” message only holds water because they have been able to get every CDN on the planet, for the most part, to participate. Conviva has a very strong offering in the video space but as that space has evolved they may find that the needs for such a specific and very expensive service dwindles. I’ve written about this before saying that I don’t believe Conviva can survive on their own, as a stand-alone service. CDNs like Akamai are now offering a competing platform to Conviva, and it’s bundled within a larger ecosystem that is much easier to sell and is a better way to buy it.

The strongest followers in the space are Dyn, DeepField, Catchpoint and New Relic, and it’s interesting to note that two of the four come from the APM space (Application Performance Monitoring). Time will tell if the APM tool companies can compete, especially when they have Cedexis giving away its monitoring product for free. Cedexis does this so that they can aggregate or pool the community data to inform their Global Load Balancing solution (the higher value product), but it’s still hard for any vendor to compete with free. I’d be interested to hear from customers who use these solutions what they like/dislike most about them in the comments section.