http://www.strom.com/awards/77.html
Over the years, I've tested hundreds of network-related computer products and built a series of test labs around the country for various clients and publications. I guess I thrive on looking at the new, the unfinished, the latest beta. Many of you have asked me for advice on setting up your own lab, and so let me share a few secrets and a few products that can help.
Setting up test labs is more than just buying a bunch of computers and cables: you have to understand how the products you want to test are used in the cold, real world of what I call userland. Any lab is an artificial world. A good lab can mimic what eventual customers are doing and demonstrate some of these end-user issues.
My first recommendation, then, is to get to know your customer's configuration and his or her mindset: what protocols are running, which operating system(s) are typical, and what is the mix of other applications on each desktop that might interfere with the product you are trying to test.
Second, don't get hung up on testing performance. While it is nice to offer the fastest product, everyone has his or her own definition. Best to skip performance issues if you can, or at least come back to them later when you really understand the interactions with other products.
Finally, get the proper tools to make it easier to set up and tear down tests. One of the biggest challenges of a lab is being able to change test environments quickly: if you have to spend all day re-installing Windows your lab won't be very productive.
There are three network-related tools that I recommend: Ganymede Software's Chariot, Keylabs' LabExpert, and Segue Software's SilkTest. Each one has its merits. None are simple to setup, but once you get them going they are relatively easy to use.
Chariot is used to measure end-to-end network performance in a variety of near-real world situations. LabExpert is useful for setting up a series of new computers with similar applications and operating system configurations, as well as running scripts that can automate other tasks. And SilkTest is useful for measuring web server performance.
Chariot comes in two different pieces: a client piece (they call it an endpoint) and the server piece, called a console, which runs on either NT or OS/2. Reports can be viewed in a browser as well as printed out.
The endpoint is a small program that is installed at the end-user desktop, and used to send acknowledgements across the network. This is how Chariot measures network throughput: the server runs through a particular scenario (say a file transfer using ftp). It sends the file to this end-point program, and in short order you can simulate what a large collection of clients can do to your network, say, if everyone downloads a 78 megabyte MPEG video when they come in to work at 9 am. You can set up various protocols and scenarios using the many templates that come with the product, and the reports are very easy to interpret.
LabExpert handles a different situation. Let's say you have a lab with 10 machines. The boss comes in, says he wants to test out the latest Service Pack from Microsoft to see if it breaks any of your existing suite of applications.
Before LabExpert, you'd have to first make sure that your setup is consistent: you might need to first reformat all the machines, then reinstall the Windows OS manually, and finally load the service pack on each machine one by one. This could take hours, not to mention being tedious. With this product, you set up a server that contains disk "images" of the particular desktop configuration you want to test: once this is done, the product transfers the image to all your lab computers in just a few minutes. There are other useful automation routines to control how a machine boots and starts up. I remember the first series of tests I did for PC Week about ten years ago: we had to have runners hitting the keyboards of bunches of machines when we wanted to do tests. LabExpert gets around all of this.
Finally, there is SilkTest. This is for automated testing of actual web pages, running in actual browsers. You can set up scripts that will navigate through a web site, run Active X controls, and do anything that a normal user would do in the course of surfing through the site. You can run in different browsers and see what breaks. This is particularly useful in testing dynamic web sites. I have not used this software yet, although when I saw a demo it looked fairly solid.
All of this software isn't cheap and will take some time to get setup. But if you want to run a lab, you'll eventually need all three products.
My report on eCommerce for Decision Resources is finally in print. I have a limited number of paper copies, and unfortunately can't post the entire text on the web. If you send a SASE with 55 cents postage to me at the address below, I'll get you a copy.
My latest review for Infoworld this week is on Sun's Java Web Server: a complete web server written in Java.
David Strom
david@strom.com
+1 (516) 944-3407
back issues
entire contents copyright 1997 by David Strom, Inc.