Dish Network's Internal Systems Are So Broken Some Employees Haven't Worked In Over a Day 46
An anonymous reader quotes a report from The Verge: Since Thursday morning, Dish Network has been experiencing a major outage that's taken down the company's main websites, apps, and customer support systems, and employees tell The Verge it's not clear what's going on inside the company. The company's Dish.com website is completely blank save for a notice apologizing for "any disruptions you may be having" while promising that "teams are working hard to restore systems as soon as possible." The Boost Mobile and Boost Infinite sites display a similar message. When we called each brand's customer support lines, there were no humans on the other end -- each call automatically hung up after delivering a recorded message about the outage.
In an ironic twist, the outage started around the time that Dish was set to release its earnings for Q4 and fiscal year 2022. CEO Erik Carlson addressed it during the company's earnings call, saying the company was experiencing an "internal outage that's continuing to affect our internal servers and IT telephony." While Carlson claimed that Dish, Sling, and the company's wireless networks were operating normally, he admitted that "internal communications, customer care functions, Internet sites" were knocked out. Internally, frontline employees have been kept in the dark about what's going on. Two sources tell The Verge that they are being told to stand by for information from their leadership teams, which haven't yet been forthcoming. They say it hasn't even been made clear whether they'll be paid. Employees have also been told that they won't be able to connect to their VPN, keeping remote workers from logging in to work.
Despite Carlson's comments that Dish's services should be working normally, Downdetector shows an increase in reports of issues using Dish Network's services, which include satellite TV and Boost Mobile's wireless network. Customers are reporting on social media that they're unable to activate new equipment or SIM cards received from the company, and alleged technicians say they can't complete installs and upgrades for customers. Customers have also said that the outage is preventing them from paying their bills. Some of the company's sites, like dishwireless.com and launch.5gmobilegenesis.com, are currently completely down and don't even display an error message. The good news is that the outage doesn't appear to be the result of a cyberattack, according to The Desk, though Dish likely hasn't concluded its investigation yet.
In an ironic twist, the outage started around the time that Dish was set to release its earnings for Q4 and fiscal year 2022. CEO Erik Carlson addressed it during the company's earnings call, saying the company was experiencing an "internal outage that's continuing to affect our internal servers and IT telephony." While Carlson claimed that Dish, Sling, and the company's wireless networks were operating normally, he admitted that "internal communications, customer care functions, Internet sites" were knocked out. Internally, frontline employees have been kept in the dark about what's going on. Two sources tell The Verge that they are being told to stand by for information from their leadership teams, which haven't yet been forthcoming. They say it hasn't even been made clear whether they'll be paid. Employees have also been told that they won't be able to connect to their VPN, keeping remote workers from logging in to work.
Despite Carlson's comments that Dish's services should be working normally, Downdetector shows an increase in reports of issues using Dish Network's services, which include satellite TV and Boost Mobile's wireless network. Customers are reporting on social media that they're unable to activate new equipment or SIM cards received from the company, and alleged technicians say they can't complete installs and upgrades for customers. Customers have also said that the outage is preventing them from paying their bills. Some of the company's sites, like dishwireless.com and launch.5gmobilegenesis.com, are currently completely down and don't even display an error message. The good news is that the outage doesn't appear to be the result of a cyberattack, according to The Desk, though Dish likely hasn't concluded its investigation yet.
"Haven't worked in over a day" (Score:2)
They are WFH, therefore it's more than "some employees" and much more than "a day".
Re: (Score:1, Informative)
"When we called each brand’s customer support lines, there were no humans on the other end "
LOL. It's been that way, with all companies, for a long time. And if you do get an actual person, it's Apu in India or some other shithole country.
Re:"Haven't worked in over a day" (Score:4)
Not all. Some of us have erratic home network connections. Others actually run those cloud server installations. Some of us just need to get away from our children to get anything done.
Re:"Haven't worked in over a day" (Score:4, Insightful)
Jokes on them because I pretend to work at the office too!
Re: (Score:2)
wank
Re: (Score:2)
Many Ukrainians haven't "worked in over a day."
Much over.
Perspective is important.
Re: (Score:2)
Website not blank (Score:3)
Re: (Score:2, Offtopic)
Re: (Score:1)
Re: (Score:2)
No, it's not fine. It's a static page with a script on it.
Re: (Score:1)
Re: (Score:2)
Two weeks later, yes the static parts of the website are up, but important functions (including the "Forgot my PIN" login flow) are still broken. And calls to the main Customer Service number listed on the website get a short mesage "We appreciate your patience as we work to upgrade our Customer Care systems as soon as possible..." (in English and Spanish), followed by hangup.
(I am not a Boost customer, but I have a close relative who is. We'll be switching her to a different carrier as soon as we can fig
This is why ... (Score:2)
... I tell our clients to use commercial hosting instead of internal servers.
Then again, for a telecommunications company, kinda embarrassing I suppose ...
Re: (Score:2)
That's what your cell phone and data plan are for. I've certainly used that when my company had an outage of their primary uplink.
Re: (Score:1)
... I tell our clients to use commercial hosting instead of internal servers.
Nonsense. When you use commercial hosting you are completely dependent on them doing things properly, and when they turn out to be incompetent fuckwads, you're screwed.
Using your own servers is preferable and better in every way, but only if you hire good people who know what they are doing. If you don't hire good people who know what they are doing, that's YOUR fault, and has nothing to do with internal servers vs. commercial hosting.
Re: (Score:3)
Frankly, that is a pretty ignorant comment. Sure, if you can afford
Re: (Score:2)
...Almost no private company is going to be able to survive major DDoS attacks. Doing so requires, in addition to a talented team, a network infrastructure costing an absolute fortune.
Or, given the obvious lack of companies spending an absolute fortune on IT, an understanding of how to measure risk to properly mitigate it.
DDoS is the nuke of cybercrime. You're either big enough to protect yourself against that, or you're not. There is no pretending you're going to defend against that, but you can spend an absolute fortune assuming it's going to happen tomorrow and again next month, and hope the business makes enough money to offset the paranoia.
Re: This is why ... (Score:1)
Cloud is no silver bullet. (Score:2)
Look you can host your stuff on AWS or Azure or whatever, but if you don't know what you are doing, you can still F**K IT UP. They provide the servers, but they don't set them up for you. We use Open Shift in AWS... if something is not configured correctly in OS and a system is down, it's our fault not AWS's fault. AWS can have issues.. but we haven't had an outage because of AWS in the 18 months we've all our stuff there.
Cynical explanation (Score:5, Interesting)
For a more conspiratorial explanation: They probably downsized a particular engineer that kept everything working perfectly, and now with him gone, something broke and nobody knows how to fix it.
Re: (Score:2)
And then the rest of the department is fired.
Not able to know key personnel means that it's a company about to go belly-up.
Re: (Score:2)
Indeed. Either keep key IT people reasonably happy or die. There is really no alternative these days.
Re: (Score:2)
Indeed. Either keep key IT people reasonably happy or die. There is really no alternative these days.
Uhhh, wouldn't that be a factual statement for pretty much all of the employees?
When the people who fix the clogged shitters walk out of the building and force the CEO to shut down the manufacturing floor due to regulations around maintaining a commercial space, the company figures out real quick how wrong the assumptions of "key personnel" are.
Re: (Score:2)
Depends on the business. Key people you always need to keep happy, but key IT people are typically much harder to replace than others and need longer until they start to be productive. For non-key people, it really depends. For your example, you can buy that as a service and do so on short notice. Yes, that is more expensive, but you can fix the problem with money.
Re: (Score:2)
...key IT people are typically much harder to replace than others and need longer until they start to be productive...you can fix the problem with money.
First off, no they're not. And that response is based on exactly how most businesses treat their "lowly" IT nerds.
And the reason that assumption exists in business is due to your latter comment. No one has to show a profit in IT. This is why every degrading assumption regarding the (non) value of IT staff is merely swept under a very expensive consulting rug.
Re: (Score:2)
Actually, they are. At least if you care about things continuing to work. I have seen several IT landscapes that got pretty much ruined by stupid decisions about replacing key IT people. Yes, it takes a few years for technological debt to pile up after such a stupid decision, but it reliably will. And at some point you got an IT that is in a state that threatens company survival. I have seen that as well. I am well aware that today's MBA morons are not able to do strategic management and that does work shor
Re: (Score:2)
Managers are 13 on the dozen, good managers are like 10%, acceptable managers are 50% and 50% are various level of bad. (Yes, I know that the percentage don't add up.)
I'd rather take a manager that's busy with stuff that I don't work with than a manager that thinks that he knows better than I regarding how to do my job.
Re: (Score:2)
Indeed. And management education has not done anything to fix that but seems to have made it worse with things like the MBA. There is a new trend called "evidence based management", so maybe they will get it right eventually (and stop graduating people that are worse than not having a manager) but it will still take time. Management incompetence is probably the largest problem we have economically these days.
Re:Cynical explanation (Score:4, Interesting)
Having had to work with Dish Business before, and knowing a former employee at Dish, it seems their problems are much deeper and longstanding than a recent firing.
In 2018 or 2019, they moved to some kind of new systems internally. One they had been working on for years, probably close to a decade. The day they finally transitioned it was a total shitshow. All their support went down on the consumer and business sides, for 3 solid days. No new orders were being processed. It looked very much like what is going on now, from the outside at least.
Their management is crap all the way to the top, their CEO is notorious for playing chicken carriage contracts for their TV programming, and getting smacked down. At any given time there will be dozens of channels that are blacked out on Dish at various localities. Sometimes nationwide. Famously they couldn't provide HBO to anyone for about 2 years. It made a lot of problems for the company I worked at, since we had contracts with our clients guaranteeing a certain channel lineup, and we couldn't give them any information as to why the channels were out or when they would be back, other than "Charlie Ergen is throwing a hissy fit."
Their satellite service has been on the way out for a long time, but now that they own Boost Mobile? I don't think people will be as tolerant of outages when it comes to keeping their phone or internet turned on.
I hope some good reporting eventually comes out on what the problem is, I'll be looking for it on my own in the coming months even if Slashdot forgets about it.
We're very quickly entering an Idiocracy world (Score:2)
Re: (Score:2)
You forgot that they may even be dead or suffering from Alzheimer's.
Re: (Score:2)
The person that manually enters new DNS entries has been out with the flu for a few days.
Re: (Score:1)
I just dodged that bullet (Score:2)
Re: (Score:2)
One annoying issue I ran into with those bargain cell resellers is one of them flat out did not support sms pictures, only text. It wasn’t until I complained did they admit it. It was Black Wireless.
Re: (Score:2)
One annoying issue I ran into with those bargain cell resellers is one of them flat out did not support sms pictures, only text. It wasn’t until I complained did they admit it. It was Black Wireless.
Running a text-only SMS service, is akin to running a pager business today.
I'd love to know which mafia owns that company. The fuck else are they still in business for.
He probably just had no idea how to setup the APN. I'm sure their support was terrible though too and weren't able to explain it.
Part of Ting Mobile also affected (Score:2)
Dish bought the T-Mobile side of Ting Mobile's business. The Verizon side still belongs to Tucows.
Since customer service starts on the Dish side of the house, CS was affected.
Ransomware (Score:1)