Yea, a1 is the one that's probably the most annoying. I get regular (daily) data from this server. I have monitoring agents in place to notify me if the system goes off line or even has a bad afternoon.
"Has this ever worked?"
Are you fucking kidding me? Here are the tcpdumps from the system showing the packet loss. Here's the output of ifconfig showing the interface as down. Here's ethtools showing no link. Here are the configuration files from yesterday showing it was on line yesterday. Here is the monthly consolidation file showing it's been configured for years. The shit I gave you when I opened the ticket in the first place!
"Did something change in the data center?"
Nothing we've done. It's a remote site and we don't manage the data center. Maybe it's a cable that went bad, someone tripped over, a switch configuration change due to a reboot, a firewall change?
Data Center Manager: "In my experience, cables don't go bad. Are you sure the system was working yesterday? I tend to believe the Networking folks over you sysadmins."
Ultimately it turns out the cable went bad. Replaced the cable and all was well again.
But damn, "was it working yesterday" really is a pisser.