D-Link Firmware Abuses Open NTP Servers 567
DES writes "FreeBSD developer and NTP buff Poul-Henning Kamp runs a stratum-1 NTP server specifically for the benefit of networks directly connected to the Danish Internet Exchange (DIX). Some time last fall, however, D-Link started including his server in a hardcoded list in their router firmware. Poul-Henning now estimates that between 75% and 90% of NTP traffic at his server originates from D-Link gear. After five months of fruitless negotiation with a D-Link lawyer (who alternately tried to threaten and bribe him), he has written an open letter to D-Link, hoping the resulting publicity will force D-Link to acknowledge the issue. There are obvious parallels to a previous story, though Netgear behaved far more responsibly at the time than D-Link seem to be."
Never buying D-Link again! (Score:2, Interesting)
wrong easy fix. try this... (Score:5, Interesting)
on date X, send bogus packets in response... not just wrong time, but seriously wrong time, like a packet with time of 9s in all fields, which would be most seriously wrong.
hopefully, it would lock up the offending junkpiles, and clear the problem right smartly.
the general idea in engineering an end to these things is to find a way to blow up the crooked machine by a seriously wrong entry that will screw up the internals. since they took an ugly and cheap shortcut by using firmware tables, they probably don't error-check their inputs from NTP and other services. so there should be a memory jump and a crash in those pirate boxes someplace.
and that puts the onus back where it belongs, on supercheap designers for obnoxious companies that don't give a shit about network etiquette. the market will punish them. that's how it should be for slap-happy outfits.
D-Link is just a bad net citizen (Score:5, Interesting)
Re:Moochers (Score:5, Interesting)
That being said, D-Link has acquired quite a bad reputation in my book. The last time they were prominently mentioned on Slashdot was when their routers were randomly silently redirecting a small chunk of HTTP traffic to D-Link advertisements, and causing the obvious mayhem in non-human-readable HTTP traffic.
I'm also wondering just how much mayhem this guy could cause on various networks by playing with the time he returns. I'm not advocating that...I'm just pointing out that D-Link is rather leaving the owners of their routers open to whatever he chooses to do to them. Adding NTP support to a product is one thing -- hardcoding it to reference an NTP server that you can't guarantee is trustworthy is another thing. Suppose, for instance, this guy drops the name due to the expenses and someone else picks it up...
To be blunt, buying D-Link hardware at this point means that you're kind of, well, asking for whatever the hardware does to you.
Wasn't this already patched? (Score:2, Interesting)
Re:Im confused (Score:5, Interesting)
* To keep the network working, the NTP system is tiered. Anything other than a time server used to redistribute time to other machines should probably access a Tier 3 system, or a Tier 2 if that is not possible. It should never hammer a Tier 1 -- this can screw up the rest of the NTP network.
* There are large lists of NTP servers, and they list access restrictions. As pointed out in the letter, this guy explicitly stated in his access rules that this server was not for client use.
* As pointed out in the letter, this guy explicitly stated in his access rules that this server was not for use outside of Denmark.
You may not be used to this sort of thing, because no such set of agreements exists for, say, webservers. However, in the NTP world, network administrators respect these, and it is why the time system continues to work.
What D-Link is doing hurts all Danish NTP users, and freeloads off a volunteer (D-Link is selling the product and profiting from it -- let *them* handle the traffic and factor any bandwidth costs into their product cost). It opens their product to potential abuse if the server becomes malicious (a properly-designed router would allow the user to specify an NTP server, or if the user is unable to configure a router, to do what the letter suggested and use a D-Link-controlled name.). It violates agreements that have been generally respected by the NTP-using administrator community for many years.
Re:wrong easy fix. try this... (Score:3, Interesting)
The market has no mechanism for punishing them. It is completely helpless to deal with this. It takes a sysadmin from a left-socialist country to deal with the things the market cannot.
send NTP replies with very low IP TTL (Score:1, Interesting)
They'll eventually stop if they don't receive any answer.
D-Link Business Development (Score:5, Interesting)
Ok, let's do some good. Are we slashdot, or what?
D-Link Business Development and Strategic Partnerships, E-mail: bdm@dlink.com
>>>
To whom ever it may concern:
Hello.
I just learned of you companies notably persistent inability and unwillingness to deal with a serious design flaw in a growing range of your products. This flaw is severly disrupting internet services for a large amount of internet participants and even though you have been informed in detail of these effects your products are having, you have done nothing of substance to resolve the issue and compensate for the damage done.
Until I learn that the issue described in the open letter do D-Link, available under http://people.freebsd.org/~phk/dlink/ [freebsd.org], was resolved in a professional and mutualy satisfying manner I will not purchase any D-Link products and will strongly discourage anybody asking for my expertise as a professional in the IT field from buying D-Link products or from engageing in any sort of business relationship with D-Link.
Sincerely
An Internet User
Mistakes in this one? Please post corrected version below and then add a 'mailto' link to the address.
Grammar Nazis, it's your turn!
Re:Moochers (Score:3, Interesting)
...or does what I'd do, and find out if any NTP replies can crash DLink's hardware. Move my real NTP server to a new IP and hostname and start advertising that, then start serving bad packets on the old address.
DLink might be more interested in fixing the problem if 75% of their hardware was returned each month for random failure.
Path to Justice (Score:5, Interesting)
2. Take a collection from the
3. Wait a month for all the legitimate users to switch to a new URL.
4. Fire up a server at the old URL reporting Midnight, Jan 1, 1900
5. Let D-Link deal with users accusing D-Link of failing to sell a Y2K compliant product in 2006.
Write DLink and let them know what you think! (Score:1, Interesting)
customerservice@dlink.com
webmaster@dlink.com
analysts@dlink.com
sale@dlink.com
si@dlink.com
broadband@dlink.com
bdm@dlink.com
edusales@dlink.com
oem@dlink.com
productinfo@dlink.com
hr@dlink.com
Re:Couldn't they filter (Score:3, Interesting)
Pot, I'd like to introduce you to Mr. Kettle.
Try pinging "pool.ntp.org". Now you now what the hell the GP babbled about.
The NTP server in question does not (so far as I know) participate in the open NTP pool, but that fact differs drastically from saying "There's no such thing as an ``NTP pool`` that can ``re-route`` anything".
And if he renames his server, he just breaks it for the people who are supposed to be using it.
"Gee, I have to PAY 80% of my bandwidth cost to let an abusive user keep using my FREE service". Something there doesn't quite sound right, eh?
I don't really see the problem with just changing the address, and in his situation, I don't think I would have even bothered trying to contact D-Link about the issue - I'd just make the change email the users that asked permission (proper NTP-etiquette says that you should always ask first, though server admins almost never turn anyone down), and leave it to the users to change over). It doesn't matter if he has 10 or 10,000 users - It only takes about 15 seconds to change one entry in an ntp.conf.
For an example, I keep my masquerade box sync'd as a stratum-3 to a dozen timeservers, and every now and then, one will change. If the admin emails me, I just update my list; if not, a few months later I might notice that one server has stopped sending me data and I pick a new one. Not the end of the world - Not even enough of a problem that I even notice it except by pure chance. And unless all twelve went down without me noticing, NTP will intelligently just use the ones that do still respond (and even if they did all die, NTP learns your machine's hardware drift well enough over time that you'd still probably stay accurate to within a few seconds per year).
No... (Score:4, Interesting)
open specifications are still the property of the creators. (kinda like the GPL)
they are licensed to 'the world' to use, so long as the specification is followed.
the spec in this case, includes disallowing certain services to certain levels of useage
So, the creators of NTP spec can (in an extreme beyond all belief example)
deny d-link further permission to use NTP at all.
Further, if they are not following the spec (honoring requests by the NTP server not to be used
in this manner) you could as the owner of one of the devices(one again, extreme example)
sue d-link for advertising/listing on the box of the products in question,
for saying they are ntp capable- when it's proven they are not compatible with the spec.
(the spec that includes respecting requests not to be used in this manner)
what are your damages? at least the cost of the affected hardware.
This is the problem... (Score:1, Interesting)
For example, my last D-Link wireless router was not made by D-Link. It was made by a Taiwanese outfit called Amit. The exact same products were sold under names varying from SMC, Asante and GVC to 3Com, US Robotics, and doubtless others as well.
The moral of the story: *most* of these manufacturers sell the exact same junk, with the exact same firmware, coded by the exact same people - just with some different logos slapped onto the chassis and the web interface. The only value in buying from a particular vendor is because of their support options (if any), or because their price is lowest. There's no differentiation in the actual feature set of the products. (Heck, for some time I ran my router on "somebody else"'s firmware because they were first to get a bug fix out).
Re:List of Affected Products: (Score:1, Interesting)
tick.usno.navy.mil ?
ntp.alaska.edu ?
montpelier.ilan.caltech.edu ?
time-b.nist.gov ?
ntp.nasa.gov ?
Are these servers (and many more) legal targets of time requests from consumer D-Link products?
Re:WTF??? (Score:5, Interesting)
He discovered a problem.
He contacted the company causing the problem.
He explained the problem, and simply asked them to fix it.
They didn't.
They put him off.
They threw a lawyer at him to threaten him.
They offered 'compensation' that didn't come close to covering his costs.
He was trying to do it all quietly and nicely, not crusading, and they wouldn't have it.
So instead of going through the often extremely troublesome and lengthy legal procedings (which are even worse than normal since this is an international case), he was hoping to publically embarrass the company into fixing the problem they caused. Seems like a reasonable attempt at a speedy solution, not a crusade.
Easy answer, boycott D-Link (Score:2, Interesting)
I just sent them the following email:
"I am a networking consultant, Cisco certified, and I talk to a lot of people about home wireless networking. I will not recommend D-Link products and today will begin actively campaigning against them for the unethical access and trouble that you have given to the GPS.dix.dk NTP server. When you have patched your products and made amends to the owner of the NTP server, then I will consider recommending your products again."
Their feedback link is on the bottom of their index page.
Letter to *MY* ISP (Score:3, Interesting)
Subject: D-Link Abuse of NTP: Action Requested
I'm certain that most of the technical staff at speakeasy reads slashdot, so you may have seen this before, but please take a peek at:
http://people.freebsd.org/~phk/dlink/ [freebsd.org]
It would make me very proud to be a $ISP customer if $ISP were to redirect *all* ntp traffic pointed to GPS.dix.dk were redirected to pool.ntp.org (or some other round-robin ntp alias). Although D-Link really needs to step up to the plate and do the right thing, I think that this would be an excellent way to lend a hand to somebody providing core internet services for free.
I'm certain that a good portion of your customer base uses D-Link equipment and any load that can be taken off of this poor guys host will be appreciated. Additionally, if a press announcement is made by $ISP about provding some relief for this guy, it will draw attention to the problem, and possibly other ISP's will follow suit.
I thank you in advance for your consideration of this issue and am very glad to be a customer of $ISP. I know if I were writing this support request to a Bell company or some other type corporation, it would fall on deaf ears at best.
-$ISP Customer
Re:wrong easy fix. try this... (Score:3, Interesting)
Re:Poul-Henning clarifies (Score:3, Interesting)
Looks to me like someone is covering tracks.
Here's what I'd do (Score:3, Interesting)
He's correct that performing complex packet matching on a Cisco router would load it too much - they just don't have the CPU to do that function for any significant traffic load.
I would configure the switch that the NTP server is on to have a SPAN port - a port to which all traffic is copied. Most Cisco switches will do this without any problem. On that SPAN port, connect a Linux box with a bit of CPU power - 2GHz would be tons. On the Linux box, setup tcpdump to match the packet patterns that D-Link routers are sending ( from TFA he has this as detected by a network consultant ).
From the output of tcpdump, extract the source IP addresses. A fairly small perl script would probably do it. Take these IP addresses and massage them into access-lists for the upstream router to block, again perl or TCL/Expect would be reasonable tools. Routers are good at blocking large lists of IP addresses - its not such a load for them as the list gets compiled and pushed onto the hardware. Depending on his router model a few thousand ACL lines would be fine.
Alternatively, he could use the same approach to detect the non-D-Link source IPs - permit these and block anything else. From his stats of legit -vs- D-Link sources this would result in a shorter access list.
The only issue here is that a D-Link behind a shared-NAT'd IP address would result in that address being blocked, but there shouldn't be too many of these. And legally he can block anything he wants - his service has no written guarantee to he should be legally safe (yeah, IANAL).
To keep costs and time down, he can probably get help from the local University ( a cool project for any CompSci students ) to do the code and Linux setup, or help from the local LUG - I'd bet there would be plenty of volunteers to set it up, and I could imagine it being done within a couple of days.
Kerry
Re:Blacklist time (Score:3, Interesting)
What do you do when every networking company carries on like that? You can't boycott them all, right? I can, and I will. If I have to do it myself to get decent equipment, then that's what I'll do. But there's no need. There is a fair amount of decent stuff out there. You just have to hunt for it. Recently, I bought a new router/hub/firewall. Took me 3 tries to find one that was acceptable. It's annoying to have to wade through product reviews, keep an eye on whatever you get for the first few days to be sure it's working right, and return the bad stuff, but there is enough crap out there you have to do it. Buying and returning bad products hurts them more than a simple boycott.
BTW, if you're curious, the acceptable router was an SMC 7004VBR, and the bad ones were a Linksys WRT54G and a Trendware-- I forget the model number, but it had an extra feature, a USB printer port.
The Linksys was especially disappointing after reading all the rave reviews in favor of it. Linksys really spoiled the WRT54G when they changed from version 4 to version 5 at the start of the year. There was one other bad thing that they fixed in spades. Older firmware versions would get you banned from dyndns.org for abuse. Very similar to what D-Link is doing to this NTP server. Not only did Linksys fix it, they went to the trouble and expense of getting dyndns.org to certify the WRT54G. Most routers just use dyndns; they don't bother with certification. Compare that to D-Link's behavior over this NTP problem. Too bad version 5 of the WRT54G was such an otherwise poor router. I'd try Linksys again sometime because their handling of their dyndns problem shows me they're trying to improve, and they do have the reputation of being the best at the wireless networking.