Tracking The Status Of Popular Websites? 104
An Unnamed Correspondent asks: "I am one of those that decided, a long time ago, to use a @yahoo.com free mail account as a permanent address, since I move from one ISP to another all the time.
Last week, I was having huge problems with my free account, the SMTP servers for Yahoo! were down. I didn't know if this was a problem local to me or if it was Yahoo!'s fault. I sent them an e-mail asking about this, but I received no reply. I have been browsing all of Yahoo! to see if they have some kind of net status, to no avail. The other day CNN.com was not working for me. Maybe it was overloaded because of the elections, but I didn't know, and I couldn't find out.
Is there some kind of Web page giving news about the status of the more popular Web services?" An interesting idea ... would something like this be possible to pull off in an effective way (and what would one do if the monitoring service itself is unavailable?)
Their MX hosts are acting weird (Score:1)
We are having a devil of a time getting that mail through to the Yahoo users. The preferential MX seems to change with each lookup. If you try to connect to less preferential MX host (higher number) it will either refuse the connection, time out, or drop the connection at the envelope sender.
This plays HELL with mailers that keep delivery hints like Exim. I think their system relies on a fresh DNS lookup for each delivery. It is as if they are using an F5 3DNS for load balancing or something. If you connect to the wrong host, your email does not get through.
Also, each host seems to have a gazillion IP addresses associated with it. Some of them respond, some dont.
Log entries like this are common:
2000-11-25 14:12:49 13zcjX-0000y9-04 Remote host mx2.mail.yahoo.com [128.11.68.59] closed connection after initial connection
2000-11-25 14:12:49 13zcjX-0000y9-04 Remote host mx2.mail.yahoo.com [128.11.69.55] closed connection after MAIL FROM: SIZE=5114
2000-11-25 14:19:07 13zYFa-0006gy-04 SMTP timeout while connected to mx1.mail.yahoo.com [216.115.107.17] after end of data (4068 bytes written): Connection timed out
I have no idea what the heck is going on over there but it is taking forever to get this mail into Yahoo. I wish they would fix it. I have had to build a separate machine just to handle the yahoo.com mail.
NetStat Systems (Score:1)
Netsaint (Score:1)
We use it at work to keep track of how often our hosted website and our NT servers are down. It is fairly easy to install, and configuration isn't too bad since there are lots of examples you can copy-n-paste. It is very stable, and the generated data is laid out in a fairly readable manner (definately understandable by non-techies.) You can even do email and pager alerts when a host or particular service goes down!
Re:There is at least one remote ping service (Score:1)
For more info see http://www.mae.net
Heiko
Nice try.... (Score:1)
SMTP? (Score:1)
Re:three words (Score:1)
if you just want to know why your yahoo mail account isn't responding, there's generally no need to whip out the UNIX Swiss Army Chainsaw (tm)
if what you're looking for is more comprehensive monitoring (specific services, machines, states, etc.) and most likely reporting/notification (email, paging, etc.) you're better off with some of the established systems...nocol comes immediately to mind (as does ipmon by mediahouse for you NT types).
-dk
three words (Score:1)
nslookup
ping
traceroute
now go monitor and such
-dk
This will be the first Processtree project? (Score:1)
Look at their news [processtree.com] page for information.
According to that page, 'the job is a quality-of-service monitoring system that provides real-time updates on the performance and availability of websites.'
Wonder what they'll do with the data, though
Netcraft says... (Score:1)
"... while [35]slashdot.org [slashdot.org] has seen a pronounced improvement in stability over the last year."
Draw your own conclusions.
Re:Netwhistle (Score:1)
Router Status Report Site (Score:1)
this is especially useful in Australia as seen recently, most of the country is knocked out by one cable going down. *sigh*
\\||//
--ooo00ooo--
Another way of checking a site (Score:1)
Another method to use if it is your dns server that has gone crazy is to use http://www.network-tools.com as mentioned in another post.
Re:IP Monitor (Score:1)
While we're mentioning monitoring software, I'll put in a blatant plug for NetSaint(www.netsaint.org [netsaint.org]) - a GPL'ed monitoring package for Linux and most all other *NIX OSes that I started about 2 years ago.
Anyway, detecting a "real" failure of network services (like POP, SMTP, etc) at a remote location (i.e. a popular website) is more difficult to do than most people may think. Since the remote host is serveral hops away from you, there are a number of potential breaking points in the network connection between you and the host.
Most monitoring software will tell you if the service is unavailable from your point of view, but this may not be accurate for the rest of the world. If you want to know whether or not a particular service like SMTP was really down on a remote host (and not just down/unreachable from your point of view), you'd need to have several locations from which you are doing the monitoring. Since you don't own/aren't in charge of the host in question, you probably won't be able to get a really accurate view of what's going on, unless you can place enough monitoring hosts around the 'net or peer with other people who are doing monitoring...
Distributed (Score:1)
I guess this would also include building traceroute maps for the monitored servers, so that you actually know beforehand what links are between the monitoring server and the monitored host. Of course, these will need to be updated regularly, as they can change.
Just a couple random thoughts that I have on this subject. It would be interesting to develop something like this.
-- hiro aka wintahmoot
Re:Netcraft (Score:1)
However, I don't think they use polling for uptime. A reboot would only last 60 secs, and they poll very infrequently (sub-daily). All reboots would thus remain undetected. A second piece of evidence for this is the fact that they've been monitoring my host (www.var.cx aka ion.var.cx) for only 5 days, yet know it's uptime is 36 days.
Hmm, I just noticed they failed to detect the OS (linux). Guess my isp's up to some funny stuff again...
Netcraft (Score:1)
Re:Netcraft (Score:1)
Re:Netcraft (Score:1)
Re:SMTP? (Score:1)
I've heard that sometimes it's even a bit faster than that!
Re: SMTP? (Score:1)
All too true -- but at least it increases shareholder value! If it makes them more money to deliver free mail slowly or not at all, who are we to argue
Customer service... (Score:1)
It might not be great for marketing that something is actually down... but its better than keeping your lip buttoned.
---
Re:Some useful sites for that... (Score:1)
According [whatsdown.net] to www.whatsdown.net [whatsdown.net], Yahoo.com was down for 10 hours (22. November)! Looks baaad!
Wonder if it was true...
Re:There is at least one remote ping service (Score:1)
For example, I can't get a reply from cnn.com.
Re: shouldn't be that hard (Score:1)
Monitoring Sites (Score:1)
Also I noticed that http://www.netcraft.com/ were doing some sort of monitoring of sites when you search to see what a site is running.
Graham
Re:Netcraft says... (Score:1)
Dave
Re:Permanent E-mail Address (Score:1)
webperf.org (Score:1)
I don't know if they monitor anything besides http. That's all I use it for.
http://webperf.org/
May sounds mean... (Score:1)
Yahoo's SMTP servers are very unreliable (Score:1)
I use qmail and follow the qmail mailing list. Others on the list have confirmed that yahoo is the problem.
IP Sentry (Score:1)
see it here [sympaticousers.org]
Re:SMTP? (Score:1)
Do it yourself (Score:1)
You can also find people who will forward email for entire domains for cheap or free. If you register your domain with Your Name Free [yournamefree.com] you can get free forwarding.
Or, you can do what I do now: Run your own sendmail box. I handle all of my domain's email. I never have to worry about it breaking, because I know that I can fix it. I also have no one else to blame beside myself.
Re:Netcraft (Score:1)
No, they use something different. I'm not sure what it is, since the explanation [netcraft.com] is laking the details. For instance, they're reporting the uptime for my server (>90 days) accurately, even though I entered it only a month ago.
Re:Nice try.... (Score:1)
Maybe it's the beer, but it looks to me like Idaho's post has everything to do with the article, especially since other posters have said that a monitor like this would need to be distributed to many points, just like ProcessTree apparently wants to do.
Yes, he's linking there with his Sponsor ID, but why is that so bad? And how in the world is it offtopic?
The Slashdot question is about Tracking the Status of Popular Websites and the ProcessTree project is about (as noted by Idaho) "a quality-of-service monitoring system that provides real-time updates on the performance and availability of websites."
The post has nothing to do with the article???
Re:SMTP? (Score:1)
It's even wrong. I always send my mail with smtp, which is the standard protocol for sending mail..
Re:Some useful sites for that... (Score:1)
Anyway, another useful service, when we are at it it www.traceroute.org [traceroute.org].
Re: SMTP? (Score:1)
Well, we are talking about Yahoo here, so YMMV...
--
Turn on, log in, burn out...
Re:shouldn't be that hard (Score:1)
I think this could be done quite "safely"; perhaps a Slash-type site, where people could post "stories" along the lines of "I'm getting `connection refused' from www.cnn.com - anyone know what's up?". People could then indicate whether or not they are having similar problems, and any official reports ("Power outage at the hosting company - should be back up RSN" or whatever) could be added.
There is one obvious reason they don't put status pages on their WWW sites - if there's a problem with the WWW site, how on earth are you going to read the status page to find out?!
It's a good idea, I think, and as long as you stick to simply relaying reports, you should be OK legally? (IANAL, though...)
I know this is wishful thinking but .. (Score:1)
I don't just mean for humans to read. If also, all clients supported this and could change address books ( and spam lists
Perhaps have that in conjunction with a system where everyone had 2 distinct addresses. If one doesn't work the original SMTP server will try the other before relaying an error to the client/sender.
Start writing those RFC's!
Mail servers down? (Score:1)
Re:Mail servers down? (Score:1)
Their POP3 servers also appear to be affected adversely by Outlook.
=====
I have experienced a problem sending email through outlook express. I had a file that, when attached to a message in outlook, would freeze up halfway through sending. It would do this consistently.
The "bad" part was that each time I tried to send the message, another two sendmail processes were created on the mail server. I assume that outlook was not properly closing the sockets when I aborted the send because it took many hours before the sendmail processes quit even though my attempts to send the message were long past. I was able to consistently reproduce this across multiple sendmail installations, I could've filled the process table relatively easily. It was bizarre. The file was an mp3 file (I could send any other mp3 or regular file), I ended up zipping it.
Maru
Re:three words (Score:1)
Which would basicly only tell you if the box was physicaly up, not if it was reponding to requests, which is more often the case with over loaded sits© Once web servers hit their configured maximum query limit they stop servicing further requests - that's what you should check for; as well as the actual physical up/down status - maybe you should add perl to your list©
--
Re:Netcraft (Score:1)
-------
Re:There is at least one remote ping service (Score:1)
-------
Re:In response to Yahoo (Score:1)
I've noticed much the same thing before. I admin approximately 60 mailing lists at work, with a few thousand yahoo.com subscribers in total. Most of the problems we have relating to mail delivery occur when our mail server gets stuck sending to Yahoo!'s SMTP servers, with the symptoms you describe - they accept connections fine but then hang when it comes to responding to commands.
This isn't the first time this has happened, either. Yahoo!'s SMTP service seems to have failed in some way or another at least once a week for the past couple of months now, with outages sometimes lasting 48 hours or longer. Yahoo! haven't responded to any of the mails I've sent them, or posted anything at all about these problems on their site.
Re:Mail servers down? (Score:1)
That was rude, obnoxious, arrogant and just dripping with sarcasm. Please keep it up.
.
If I can't get to a site... (Score:1)
Don't forget the looking glass... (Score:1)
Anyway, to assess the icing conditions on the Internet, you should also look at the routing. One good starting point for locating so-called looking glasses is Nanog [nanog.org].
Port 25 blocked (Score:1)
Re:Tracking? (Score:1)
One would be to start distributed sniffers on every network around the globe, that merely sent logs back to some centralized location for tracking. We could write a dozen different worms that manage to invade various OSs, and patch init so it sends data out over the network--even through firewalls!
The other easy way is to put a patch into everybody's browser that sends stats back somewhere--but then we'd need to make them upgrade to it.
I got it! Why don't we just all go IPv6 and tell the world that there isn't a browser in existence that supports it! Then they'll have to upgrade!
Oh wait, didn't M$ put tracking hooks into IE on ME?!?!!
Seems to me I already saw that..... heh
Re:Port 25 blocked (Score:1)
Aarno
Myrealbox (Score:1)
I mailed them a couple of times about this but I never got a reply. I now use an e-mail alias which seems to be much more reliable.
Disclaimer: both services were (are?) still in beta when I used them, though that status must have been for over a year.
Re:Your spam is broken even (Score:1)
Re:Permanent E-mail Address (Score:1)
But we have one. (Score:1)
It wasn't their POP3 servers, per se, but... (Score:1)
A few cents (Score:1)
Sites like Crosswinds.net [crosswinds.net] have their own updates page [crosswinds.net] where they give up-to-date information on what's right and wrong about their service. I haven't seen any other websites/services which give such up-to-date information.
On a related note, I'd like to add that managing your own mail is the cleanest solution, rather than depending on different ISPs' POP3 and SMTP servers. This of course requires a permanent connection in most cases, but is the best. This way you manage your own e-mail, and move it around, when you move. You obviously need to know how to setup and manage such a server yourself at home/office, but it pays you back.
uptime robot at arsdigita.com (Score:1)
Netwhistle (Score:1)
Re:Permanent E-mail Address (Score:1)
Since the server is under your control, you can be reasonably assured that the machine won't magically disappear for some unknown reason, and as long as you maintain the machine and pay the domain fees, you will have a permanent mailbox. You can also play with nifty things like using dynamic DNS, running a webserver, writing your own POP3 daemon that uses PostgreSQL, and so forth.
One nice thing about doing this is that you can setup relaying to allow you to relay messages through your freshly installed mailserver, and get around blocks placed on your ISP. (Some people block some parts of the big broadband ISPs, like rr.com and mediaone.net)
(As to how to get a machine attached to the network, some ISPs *do* offer free co-location space to people who work there. :)
Well... (Score:1)
Try This... (Score:1)
https://lm.lcs.mit.edu/px.html
If you cannot connect to a site and you think your local ISP is responible use the above link.
internet traffic report (Score:1)
Processtree (Score:1)
(Although he didn't have to post the link with his sponsor number in it... Exactly why I strip those numbers out when I go to look at sites like that... greedy bastards...)
Re:Mail servers down? (Score:1)
You can still send via a web interface, but that is pain when you want to keep copies.
And yes to ask question about the mail services you have to send mail. I have that conversion with them before. you want to talk CATCH22?
Note Yahoo commonly gets swamped the SMTP from outside world into Yahoo system to an actual account can take upto 8 hours. So enjoy the FREE mail.
jackb
Yahoo mail (Score:1)
GeoCities has that. (Score:1)
Re:Internet Weather/Traffic Report (Score:1)
Re:SMTP? (Score:1)
Re:Some useful sites for that... (Score:1)
eBay posts system status announcements (Score:1)
PING is not enough (Score:1)
I use the service of operamail [operamail.com]. They have been very reliable some times ago but now they have downtime at least once the week. (Hey this can not be possible. They run their servive on NT.)
You get an answer to your ping but the webmailer and the pop3 access both do not work. Just using ping to detect server outages seems not to be sufficient.
Monitorization (Score:1)
But if I am mistaken and someone has an idea, I'd be happy to pass it along to whomever I could (giving full credit of course
humor for the clinically insane [mikegallay.com]
Re:Permanent E-mail Address (Score:1)
I've got a cable modem, so it isn't a dialup....
In any event, if I use @Home's SMTP server, I have two problems. One, regardless of how I set up my clients, half of the other clients out there will respond to my @Home address, which is *not* where I want to receive my mail. Using my own SMTP server, I can set the thing up so that it looks like it came from wherever, and I will then get replies where I want them (pobox.com) and people who add my address to their addressbook will add the right one (pobox.com, *not* home.com).
The second problem is that with @Home's mail servers, I would have no confidence using their SMTP mail servers that my mail was actually getting out with any degree of reliability....
In any event, I haven't had any problem using my own computer's SMTP. If push really came to shove, I might pay pobox the extra money to be able to use their SMTP servers.
(The other SMTP available is the one for the ISP where I really receive my E-mail, and which hosts my website. This is sonic.net, a *great* ISP for anybody in the right parts of Northern California. I still use it as my emergency backup dilaup, though naturally I don't dial in to it much given the cable modem. However, I can't use their SMTP servers from my @Home account, because they have the anti-spam "we won't send mail not from sonic.net" policies in place.)
-Rob
a good idea? (Score:1)
Mail dot com sucks a LOT more than Yahoo! (Score:1)
When you press them for an answer about their lousy service, altavista/mail.com will send a form email that basically says "live with it".
Re:Internet Weather/Traffic Report (Score:2)
Another Andover site that's still around is one of the earliest online free code archives -- freecode.com [freecode.com].
Nobody ever "heard of" Andover back in those days because each of the company's original sites was treated as a separate entity and we never pushed the "network" thing. But we were there, doing our little thing and having a nice, low-key time. :)
- Robin
Monitoring service availability (Score:2)
The Blame Game (Score:2)
Possible choices:
- Your ISP
- A backbone router went dead
- Your local network admins have put up a firewall...
- You mistyped the URL....
- the site is slashdotted
- etc..etc...
I'm not so sure many of these sites would WANT to have their network outage publicly available. Might be somewhat embarassing for some...
Cheers,
Vic
try.. (Score:2)
You can also try pinging yahoo's mail servers.
Personally I think you kind of get what you pay for. I used to use hotmail at one point as I was doing lots of traveling. They LOST some of my email while upgrading there servers. They said sorry. I stopped using them.
Free email is not a lucritive busines. It makes its money through advertising. As you may have heard (or not) advertising on the web is not considered a lucritive business. Hey doubleclick took a hit this last quarter, and there stock was down. You're best bet is to talk to yahoo about a pay account. They have them and then no matter who you use as an ISP you can have access through an @ yahoo mailbox. The other option is to use another portol for free mail.
I don't want a lot, I just want it all!
Flame away, I have a hose!
Re:shouldn't be that hard (Score:2)
Re:shouldn't be that hard (Score:2)
Hosting service benchmarking (Score:2)
Tracking which site uses which provider and THEN tracking their availability and performance should give us some indication on how the big names in the hosting business serve their best known customers. This information should prove to be valuable even to those of us who don't really make the hosting companies' most-important-customers list.
Sounds like a good idea to me! (Score:2)
However, if it was done right, ISPs could install a package that sort of "mirrored" the test results, so that their customers could see how things looked at the ISP's offices.
Barclay family motto:
Aut agere aut mori.
(Either action or death.)
EVEN BETTER! (Score:2)
Just buy yourself a DOMAIN for only $15 / year at DirectNIC [directnic.com] or a number of other BARGAIN domain registrars. DirectNIC offers your domain UNLIMITED numbers of both web addresses and email addresses...
How can this be???? Because each email address/URL merely redirects the mail/URL to your chosen account/site...
SO, you can get your own DOMAIN, HOSTING, REDIRECTION, URL PARKING, and PERMANENT UNLIMITED EMAIL ADDRESSES for the $15 / year bargain.
shouldn't be that hard (Score:2)
Re:Some useful sites for that... (Score:2)
Tell me what makes you so afraid
Of all those people you say you hate
oh the irony! (Score:2)
"I sent them an e-mail asking about this..."
Re: shouldn't be that hard (Score:2)
Internet Weather/Traffic Report (Score:2)
Re:NEVER use your ISPs SMTP services! (Score:2)
Next, where do you get this additional hardware cost stuff? From using your ISPs mail server? its going to cost you to pay for more hardware? Wether you use it or not, they will have a mail server, and you WILL have to pay for it, regardless.
Furthermore, why do you not like the dial up list? Do you know how much spam that stops? Throwaway dial up accounts account for most of the spam sent these days.
Next, Users arent clueless by using their ISPs mail server, and they can run an OS that can handle mail traffic wonderfully, but WHY? they dont NEED to, and, they have the potential of getting their mail rejected because they're on a dialup (including broadband IPs)
Finally, if your ISPs mail servers accept the mail, and say it will be delivered, and it takes 'hours' for it to get there, its more than likely not their fault. Now if you said that you cant connect to your ISPs MTA, then you can blame them...
These are the views of me, a competant mail administrator of a redundant mid-sized network. I speak only the truth of my views, and is not meant to put anyone down, but offer a differant perspective.
sure its possible (Score:2)
There is at least one remote ping service (Score:3)
Re:Mail servers down? (Score:3)
The Yahoo servers for SMTP services that USER CONNECT TO have been quite flaky. I have outages that lasted up to a month. POP3 is also bad from time to time.
Their POP3 servers also appear to be affected adversely by Outlook. I have a friend who insists on using Outlook, and she e-mailed me. Yahoo's POP servers choked on it; I had to log in through their web access to delete the message. It's repeatable, though I don't know what part of the message kills it.
To top it off, she's one of these community college computer science students. She won't listen to me when I tell her that Outlook isn't standards-compliant. (And, I had to do three hours of tech support to help her install a second hard drive in her Windows 98 machine, to demonstrate what her third year at St. Lawrence College has taught her. She still doesn't have it working, mostly because she doesn't believe you can only run two IDE devices on each IDE bus.) <sigh>
Note Yahoo commonly gets swamped the SMTP from outside world into Yahoo system to an actual account can take upto 8 hours. So enjoy the FREE mail.Yeah, I use one of these accounts. I just set up Sendmail on my gateway machine.
In response to Yahoo (Score:3)
Probably the best way to determine an outtage is to do some inspecting yourself, run pings, traceroutes, telnet to the service, test, and probe to find out what could be going on.
Re:Permanent E-mail Address (Score:3)
Ironically, as I write this, the mail-abuse.org DNS servers are unavailable. Interesting...
When it comes back up though, goto http://mail-abuse.org [mail-abuse.org] for more information
Permanent E-mail Address (Score:3)
Yes, I know this is a tangental point to the article, but...
For a permanent E-mail address, consider a forwarding service, e.g. pobox.com. That's what I use. It's not free, but it's only something like $15/year. The advantage of this is that you can keep the same E-mail address always (or at least for as long as pobox.com survives), even as you move between accounts. If your free hotmail or yahoo account starts to go south, get one somewhere else... and just point your pobox.com forwarding address at the new one.
pobox.com seems to be pretty reliable; I don't think I've lost any E-mail. With few exceptions, every POP/SMTP/mail server I've used has had trouble at one time or another. (We're not even going to talk about @home.) Forwarding services are a little easier to get right. Since I have the freedom to point my "same" E-mail address at any mail server I want, I can achieve greater stability without always having to tell people to change my E-mail addres....
(I also use my own computer's sendmail to *send* mail, instead of an external SMTP server. Hey, why not? It's not like I have sendmail running, I just give it a one-time invocation. I've set it up so that the From address is my pobox.com address.)
-Rob
Some useful sites for that... (Score:5)
Netcraft will tell you the uptime history of any site that it watches (currently over 22 million)
Keynote: internetpulse.keynote.com
Keynote has some really cool cross-backbone nodes that tell you the performance from one backbone to another.
Whatsdown.net: www.whatsdown.net
Whatsdown watches various sites and backbones and tells you the current performance of them. This is probably what you want, because it watches specific popular sites.
Check out Google's directory (DMOZ) for some more sites like these. I hope this helped.
-------