http://strom.com/awards/340.html
That nothing comes from violence
And nothing ever could
For all those born beneath an angry star
Lest we forget how fragile we are
(Sting, "Fragile")
Summer may be officially over, but the events
of the past month have gotten me thinking about how fragile the Internet can
be. While Sting wasn't talking about routers and networks, his song applies in
spades.
The recent attacks have gotten plenty of news
and caused lots of tumult and pain for many users and network operators. But
the story of the Internet's fragility is still being written, and there are
other troubling signs that I have been reading about. One concerns the low-end
router/firewall/hub products that I recommended people install on their home
networks in my last essay. (Some of you may not have received it because of
filtering or AOL blocking of mailing lists. You can view it on the web here.)
The issue with these low-end firewalls is that
they can create all sorts of trouble for network operators if their firmware
has bugs. There have been two reported instances with Netgear and SMC products,
and I am sure that there are others that haven't been publicized yet.
Before I tell you about the bug, I first have to give you some background information about the Network Time Protocol or NTP. Computers, routers and other Internet-connected devices use NTP to keep track of the correct time. This may seem like no big deal, but having the correct time is important in the world of computing. Your files are created with a date and time stamp, so, if your clocks are off, you can't compare the most recent versions. Various server processes create log files: Without the correct time those logs are meaningless. Routers use time stamps to filter or block certain sites, and e-mail messages are sorted in most of our inboxes according to the time they are sent. (I found out about this the hard way when I sent out one of my essays from my Dad's home computer that had the wrong date on its clock. Needless to say, that PC had a bad battery and wasn't using the NTP protocols.) You can find more information about NTP here.
Anyway, NTP was created to make all this
time-keeping easier. Like other Internet protocols, it has a server piece and a
client piece. The servers, several hundreds of thousands of them around the
world, keep accurate time and are trusted to do so by their users. The client
piece is loaded on a PC or a router or other device, and queries the server
periodically to make sure that the internal time matches the server's time.
Windows XP and most versions of the Macintosh
(post version 8) support NTP to set their clocks. To check this out, go to
Control Panels, then Date and Time. For Windows, go to the Internet time tab
and you'll see the entry for time.windows.com. For Macs, click on the server
options under the Use a Network Time Server enter and you'll see the server
called time.apple.com. Both servers are provided by their manufacturers to
support this function, but many of the NTP servers are installed at either
government agencies or universities and provided as a public service by their
network operators in the old-fashioned, non-commercial spirit of the original
Internet. If you are running older Windows versions, there are a number of NTP
client programs out there that you can install (and Windows 2000 also supports
NTP but from the command line only). The good news about NTP is that once you
set it up, you don't have to mess around with it. It is one of the best
Internet applications that I know about -- small, clean and parsimonious with
its packets. The bad news is that most people don't know about it at all.
The key word in that earlier paragraph is
"periodically." Query your NTP servers too much -- say once every
second or so -- and the queries begin to look like an attack on them to alert
network operators. That is exactly what happened with a bunch of Netgear
routers here in the United States, and the SMC routers in Australia. The
Netgear routers began flooding the University of Wisconsin at Madison's network
time servers with requests, filling up their network bandwidth to the tune of
megabits a second. Luckily, their network administrator Dave Plonka was able to
figure out the problem and work closely with Netgear to come up with
remediation. His exploits are an example of the kind of network forensics
necessary to track down problems; read about them here:
Apparently, the Wisconsin NTP server was just
picked at random to service several different brands of Netgear routers
(specifically the RP614, DG814, MR814, and HP314). Plonka estimates that more
than 700,000 of these routers are currently in use and hitting his NTP server
on a daily basis. That is a lot of routers, and a lot of network traffic to
deal with.
The problem is that probably 699,990 users
think their routers are working just fine. The only issue for them is that
their clocks may not be synchronized properly. The fixes have
already been created by Netgear for all but one of the products and are
available on the company's Web site.
Here's the problem: I would guess that 99
percent of the users of these products have never downloaded any firmware, and
the process for doing so isn't all that easy. Another issue: Most of the users
of these products aren't registered, and there isn't any easy way to find them
to tell them to fix their firmware. (I have recommended these products to many
of my own friends and contacts, so that is one of the reasons I am writing
about this in detail.)
Netgear isn't the only one with badly written
NTP firmware. Owners of SMC's 7004VBR and 7004VWBR series routers probably
don't know that they are hitting a server in Australia with NTP requests. The
activity is with enough frequency that the Australian Internet providers have blocked
these routers from further connections. Australia is a bit touchy about sharing
their bandwidth, given how little of it they have with the rest of the world.
(I was unable to get any official comments from SMC unfortunately.)
We were lucky this time around: We had a
diligent network admin who took the time (and it was time consuming, tracking
this down). We had a responsible vendor, who issued timely fixes. And we had a
little-known protocol that doesn't generate much trouble otherwise and could be
easily isolated. Now imagine that circumstances are different: We had sloppy
network admins, uncooperative vendors and problems with e-mail instead of
little-known NTP. That is what is keeping me up at night, wondering where the
next outage will happen.
If you own one of these routers, please update
your firmware and get them fixed. If you are a router vendor, please work to
ensure that your first release of a new product doesn't contain hard-coded NTP
servers and follows Plonka's recommendations for building a well-behaved NTP
client into your product.
If you have noticed an improvement in my grammar and syntax, there is a reason. I am pleased to announce that Jennifer Bosavage will be my managing editor. Jen was responsible for bringing me into VARBusiness several years ago and I look forward to working with her here at Web Informant HQ.
Entire
contents copyright 2003 by David Strom, Inc.
David
Strom, dstrom@cmp.com, +1 (516) 562-7151
Port
Washington NY 11050
Web
Informant is (r) registered trademark with the
U.S.
Patent and Trademark Office.
ISSN
#1524-6353 registered with U.S. Library of Congress
If
you'd like to subscribe (issues are sent via email), please send an email to:
mailto:Informant-request@avolio.com?body=subscribe.