That nothing comes from violence
And nothing ever could
For all those born beneath an angry star
Lest we forget how fragile we are
Summer may be officially over, but the events of the past month have gotten me thinking about how fragile the Internet can be. While Sting wasn't talking about routers and networks, his song applies in spades.
The recent attacks have gotten plenty of news and caused lots of tumult and pain for many users and network operators. But the story of the Internet's fragility is still being written, and there are other troubling signs that I have been reading about. One concerns the low-end router/firewall/hub products that I recommended people install on their home networks in my last essay. (Some of you may not have received it because of filtering or AOL blocking of mailing lists. You can view it on the web here.)
The issue with these low-end firewalls is that they can create all sorts of trouble for network operators if their firmware has bugs. There have been two reported instances with Netgear and SMC products, and I am sure that there are others that haven't been publicized yet.
Before I tell you about the bug, I first have to give you some background information about the Network Time Protocol or NTP. Computers, routers and other Internet-connected devices use NTP to keep track of the correct time. This may seem like no big deal, but having the correct time is important in the world of computing. Your files are created with a date and time stamp, so, if your clocks are off, you can't compare the most recent versions. Various server processes create log files: Without the correct time those logs are meaningless. Routers use time stamps to filter or block certain sites, and e-mail messages are sorted in most of our inboxes according to the time they are sent. (I found out about this the hard way when I sent out one of my essays from my Dad's home computer that had the wrong date on its clock. Needless to say, that PC had a bad battery and wasn't using the NTP protocols.) You can find more information about NTP here.
Anyway, NTP was created to make all this time-keeping easier. Like other Internet protocols, it has a server piece and a client piece. The servers, several hundreds of thousands of them around the world, keep accurate time and are trusted to do so by their users. The client piece is loaded on a PC or a router or other device, and queries the server periodically to make sure that the internal time matches the server's time.
Windows XP and most versions of the Macintosh (post version 8) support NTP to set their clocks. To check this out, go to Control Panels, then Date and Time. For Windows, go to the Internet time tab and you'll see the entry for time.windows.com. For Macs, click on the server options under the Use a Network Time Server enter and you'll see the server called time.apple.com. Both servers are provided by their manufacturers to support this function, but many of the NTP servers are installed at either government agencies or universities and provided as a public service by their network operators in the old-fashioned, non-commercial spirit of the original Internet. If you are running older Windows versions, there are a number of NTP client programs out there that you can install (and Windows 2000 also supports NTP but from the command line only). The good news about NTP is that once you set it up, you don't have to mess around with it. It is one of the best Internet applications that I know about -- small, clean and parsimonious with its packets. The bad news is that most people don't know about it at all.
The key word in that earlier paragraph is "periodically." Query your NTP servers too much -- say once every second or so -- and the queries begin to look like an attack on them to alert network operators. That is exactly what happened with a bunch of Netgear routers here in the United States, and the SMC routers in Australia. The Netgear routers began flooding the University of Wisconsin at Madison's network time servers with requests, filling up their network bandwidth to the tune of megabits a second. Luckily, their network administrator Dave Plonka was able to figure out the problem and work closely with Netgear to come up with remediation. His exploits are an example of the kind of network forensics necessary to track down problems; read about them here:
Apparently, the Wisconsin NTP server was just picked at random to service several different brands of Netgear routers (specifically the RP614, DG814, MR814, and HP314). Plonka estimates that more than 700,000 of these routers are currently in use and hitting his NTP server on a daily basis. That is a lot of routers, and a lot of network traffic to deal with.
The problem is that probably 699,990 users think their routers are working just fine. The only issue for them is that their clocks may not be synchronized properly. The fixes have already been created by Netgear for all but one of the products and are available on the company's Web site.
Here's the problem: I would guess that 99 percent of the users of these products have never downloaded any firmware, and the process for doing so isn't all that easy. Another issue: Most of the users of these products aren't registered, and there isn't any easy way to find them to tell them to fix their firmware. (I have recommended these products to many of my own friends and contacts, so that is one of the reasons I am writing about this in detail.)
Netgear isn't the only one with badly written NTP firmware. Owners of SMC's 7004VBR and 7004VWBR series routers probably don't know that they are hitting a server in Australia with NTP requests. The activity is with enough frequency that the Australian Internet providers have blocked these routers from further connections. Australia is a bit touchy about sharing their bandwidth, given how little of it they have with the rest of the world. (I was unable to get any official comments from SMC unfortunately.)
We were lucky this time around: We had a diligent network admin who took the time (and it was time consuming, tracking this down). We had a responsible vendor, who issued timely fixes. And we had a little-known protocol that doesn't generate much trouble otherwise and could be easily isolated. Now imagine that circumstances are different: We had sloppy network admins, uncooperative vendors and problems with e-mail instead of little-known NTP. That is what is keeping me up at night, wondering where the next outage will happen.
If you own one of these routers, please update your firmware and get them fixed. If you are a router vendor, please work to ensure that your first release of a new product doesn't contain hard-coded NTP servers and follows Plonka's recommendations for building a well-behaved NTP client into your product.
If you have noticed an improvement in my grammar and syntax, there is a reason. I am pleased to announce that Jennifer Bosavage will be my managing editor. Jen was responsible for bringing me into VARBusiness several years ago and I look forward to working with her here at Web Informant HQ.
Entire contents copyright 2003 by David Strom, Inc.
David Strom, firstname.lastname@example.org, +1 (516) 562-7151
Port Washington NY 11050
Web Informant is (r) registered trademark with the
U.S. Patent and Trademark Office.
ISSN #1524-6353 registered with U.S. Library of Congress
If you'd like to subscribe (issues are sent via email), please send an email to: