Web Informant #180, 11 Decmber 1999:
Measuring web performance

http://www.strom.com/awards/180.html

We all know the web is too slow. Finding the most troublesome spots takes part detective, part engineer, and part just plain persistence. So a number of companies are doing something about at least measuring and reporting on these bottlenecks, in the hope of fixing or at least avoiding them.

I had a chance to evaluate their offerings recently. Keynote Systems and Service Metrics (recently acquired by Exodus Communications) have for-fee service offerings exclusively, while Web Partner has both free and for-fee services. Mercury Interactive has both service offerings and testing products available.

Both Keynote and Service Metrics claim similar methodologies. Each places a series of 100 or more "agents" or software monitors at various locations around the world, connected to particular Internet backbone and primary access providers. These agents send signals through these networks to a series of common web destinations, such as eCommerce, general consumer and portal sites, as well to custom destinations specified for a fee by the site's owner. The agents calculate packet delays, overall latency and other measurements to get to these sites, and then send this information back to a central repository. The company in its reports then summarizes this.

Keynote is very open about the location of its agents – with 64 in the US and another 27 located around the world. Service Metrics doesn't divulge where their agents are located. Keynote also offers free samples of various reports are available on the company's web, and visitors can try to obtain a free analysis of their own web site (although I was unable to do so). Service Metrics currently doesn't offer any free services, and has very low-res reproductions of its reports on its web. Mercury offers Topaz ActiveWatch, which monitors a site's business processes from ten different locations on Exodus' network, and can provide for fairly sophisticated scripting of how to test the workflow for these processes. I signed up for a free trial (it costs $750 a month) and received my first sales call within a few minutes, and began getting reports within a day – very responsive and impressive. Active Watch uses the same scripts that the other Mercury products use: once a customer develops a script (to navigate to a particular page on a web site, execute a particular transaction, etc.), it can be used across the board in many different products and services.

Keynote's Perspective service costs range from $295 to $995 a month. This puts them out of the reach of the average sized web operator, and is on par with the access line charges for a typical business T-1 circuit. They also offer a stripped-down single-city Lifeline service for $695 per year, although this isn't really a very effective price point. Service Metrics has more reasonable fees than Keynote, ranging from $295 to $495 per month. ActiveWatch costs $750 per month.

There are two other offerings, one lower-cost and one higher-cost. Webpartner.com offers Secret Shopper Checkout that monitors the checkout page every 15 minutes of a specified storefront ($349/yr). They have a free service that monitors basic site availability and will email you weekly status reports that I've used over the past few years. Manage.com offers a complete eCommerce package, starting at several tens of thousands of dollars, with in-depth monitoring of various processes.

The tricky part for each of these companies is measuring the performance of complex sites. Consider the following:

Without careful scrutiny of the methods used to produce each vendor's reports, it is hard to say exactly what each company is measuring. If you are interested in purchasing one of these products, I urge you to examine the reports produced and understand the kind of measurements you are getting for your money. And good luck keeping your site up.

To subscribe, send a blank email to
webinformant-subscribe@egroups.com

To be removed from this list, send a blank email to
webinformant-unsubscribe@egroups.com

David Strom
david@strom.com
+1 (516) 944-3407
back issues
entire contents copyright 1999 by David Strom, Inc.
Web Informant is ® registered trademark with the U.S. Patent and Trademark Office.
ISSN #1524-6353 registered with U.S. Library of Congress.