Salesforce Triggers 15-Hour Shutdown After Faulty Script Starts Granting View/Modify Access (zdnet.com) 29
Friday Salesforce "was forced to shut down large chunks of its infrastructure," ZDNet reports, calling it one of the company's biggest outages ever:
At the heart of the outage was a change the company made to its production environment that broke access permission settings across organizations and gave employees access to all of their company's files. According to reports on Reddit, users didn't just get read access, but they also received write permissions, making it easy for malicious employees to steal or tamper with a company's data...
Salesforce said the script only impacted customers of Salesforce Pardot -- a business-to-business (B2B) marketing-focused CRM. However, out of an abundance of caution, the company decided to take down all other Salesforce services, for both current and former Pardot customers. "As a result, customers who were not affected may have also experienced service disruption, including customers using Marketing Cloud integrations," Salesforce said.
A status update at Salesforce.com reports that the final duration of the service disruption was 15 hours and 8 minutes.
Salesforce said the script only impacted customers of Salesforce Pardot -- a business-to-business (B2B) marketing-focused CRM. However, out of an abundance of caution, the company decided to take down all other Salesforce services, for both current and former Pardot customers. "As a result, customers who were not affected may have also experienced service disruption, including customers using Marketing Cloud integrations," Salesforce said.
A status update at Salesforce.com reports that the final duration of the service disruption was 15 hours and 8 minutes.
Interesting point for me is... (Score:3)
Re: (Score:1)
...that despite being extremely difficult to customise and modify flow, rules per organisation, Salesforce has a serious market share. That much "complexity" is bound to collapse occasionally if rarely.
Oracle has a serious market share too, and their product is absolute garbage.
It's both sad and pathetic, but quite often market share is a measure of mass ignorance, not quality.
This time it seems the problem is/was? very very global...
Do you know what a public company calls getting hacked in the 21st Century? A "faulty script". Don't bother asking what the problem was.
You're not going to get the truth anyway. They have shareholders to think about.
Re: Interesting point for me is... (Score:1)
You must be a conspiracist. Oracle has more than one product, and you just do not like some of them. And many courageous companies will tell you the truth these days, but things are rarely basic and simple to explain, the readers also need to be able to read and have enough brain to understand...
Re: (Score:1)
You must be a conspiracist. Oracle has more than one product, and you just do not like some of them. And many courageous companies will tell you the truth these days, but things are rarely basic and simple to explain, the readers also need to be able to read and have enough brain to understand...
The public has been educated enough to understand "we got hacked", and it's hardly a conspiracy to find a publicly traded company bending the truth around a "faulty script" in order to protect the shareholder.
You act as if we've never seen companies behave that way before. We've seen it countless times. In fact, we've seen it often enough to expect it.
Re: (Score:1)
Its called production testing - live customers is just a bonus.
Who approved that change, and why was there no quick rollback?
For every time you are caught out, there are probably a dozen more incidents swept under the carpet.
Re: (Score:3)
Do you know what a public company calls getting hacked in the 21st Century? A "faulty script". Don't bother asking what the problem was.
You must have never went through the process of getting SOC 2 or 3 compliance if you think it would be easy for Salesforce to lie that drastically about the cause of this incident. Sweeping an incident no one knows about isn't as hard, but there will be regulators looking into this. The controls Salesforce would need to have in place to obtain SOC compliance would almost certainly not allow them to hide a security breach by calling it a faulty script. I've only had to deal with SOC 1 compliance in the past,
Re: (Score:3)
Oracle had one of the best database on the market for many years, and built a large customer base because of that.
Oracle bought one of the top ERMs on the market, and acquired and built a large customer base because of that.
Oracle bought the best CRM package on the market, and acquired and built a large customer base because of that.
Oracle bought my favourite hardware provider, and totally fucked that one up.
Oracle bought one of the two best application servers on the market, and acquired and built a large
Re: (Score:2)
...that despite being extremely difficult to customise and modify flow, rules per organisation, Salesforce has a serious market share.
Suddenly, without us ever noticing the company's growth, I notice that the tallest building in just about every major US city is the Salesforce Tower.
clearly they needs some better change control (Score:1)
and auditing before committing.
This is a beginners mistake... speaks volumes about the company.
Re: clearly they needs some better change control (Score:1)
This is why I don't commit to vendors without verifying what's running in the background. If I didn't explicitly ask for it, I don't want it.
That is why sales force has intractable problems. I stopped using sales force for anything serious or non-serious when I kept getting the white screen of death saying numbers not found.
Re: (Score:2)
clearly they needs some better change control and auditing before committing. This is a beginners mistake... speaks volumes about the company.
It takes significant arrogance to believe anyone has perfect change control and auditing which makes mistakes like this impossible. There is always room for human error, and if you don't think there is room for error in your process it only shows you have hit the limit of your creativity and imagination. That isn't a strong criticism; everyone has a limit to their capabilities. And that limit is where the room for error comes from.
One problem with applications as big as Salesforce is that the smallest appli
Re: (Score:1)
which is all brought about by patronism, nepotism, cronyism and greed
No SLA (Score:1)
A company I previously worked for started with their primary customer user interface being written in SalesForce sending data to our backend. SalesForce had a similar long outage and we had to scramble to put up something that at least our call centers could use to take orders by phone.
There was a huge cry from our sale people and our merchants about the issue and we were getting constant bombardments for "when will it be back up". SalesForce will not commit to any SLAs on outages so we no matter how much
Netflix reliability management and testing (Score:5, Insightful)
Even if video streaming doesn't work out for them, I feel like Netflix could pivot into systems engineering and consulting for the way they've built their environment for reliability and how they regularly test it by clobbering parts of it intentionally.
So many other systems out there, even ones not as burdened with the usual sense of "legacy systems", aren't built that way from the beginning. They're a tangled mess of dependencies which make change testing and fault tolerance extremely difficult and seldom possible to test.
Re: (Score:1)
Second this. However, it's easy to intentionally break things when you're the latest hotness. I'd love to have a chaos monkey running where I work, but we're too busy building new things and fixing old things. To also have stuff breaking randomly (yes, I know ChaosMonkey is opt-in) would just drive us up the wall.
Re: (Score:1)
Re: (Score:2)
Netflix have an incredibly simple environment compared to a company like Salesforce.
They have full control over the content, the customer capabilities, their change processes and the logic they execute. You try asking Netflix to give you admin access over your own account, let you upload your own films, let you change the caching rules, the licensing agreements, the encoding algorithms.
Netflix have done a great job but that doesn't mean for a moment that they could deliver something the scale and complexity
Re: (Score:2)
A fair criticism, but with something like Salesforce or many other cloud-based systems they introduce their own complexity to drive up licensing revenue, which in turn makes the system more complicated and more difficult to make reliable.
Re: What is a Salesforce? (Score:2)
Why is there always someone who can't use Google on here?
Re: (Score:2)
What is a Salesforce? In read the damn thing, and I still have no idea what it is. Must be an SAP competitor.
If you wanted to be so pedantic, you should at least tell us what an SAP is. At least you didn't mention Microsoft or Oracle, because then you would have had to tell us what those are too.
Re: (Score:2)
It's a contact management system that allows increasingly sophisticated business oriented workflow, rapidly improving elements of case management and an embedded execution engine allowing customers to supplement core capabilities with additional functionality - e.g. billing, invoicing, communications management and full scale CRM.
While not a competitor to the core SAP ERM there are a number of SAP products that compete with Salesforce, yes.