SoftBank's New AI Makes Angry Customers Sound Calm On Phone 104
SoftBank has developed AI voice-conversion technology aimed at reducing the psychological stress on call center operators by altering the voices of angry customers to sound calmer. Japan's The Asahi Shimbun reports: The company launched a study on "emotion canceling" three years ago, which uses AI voice-processing technology to change the voice of a person over a phone call. Toshiyuki Nakatani, a SoftBank employee, came up with the idea after watching a TV program about customer harassment. "If the customers' yelling voice sounded like Kitaro's Eyeball Dad, it would be less scary," he said, referring to a character in the popular anime series "Gegege no Kitaro."
The voice-altering AI learned many expressions, including yelling and accusatory tones, to improve vocal conversions. Ten actors were hired to perform more than 100 phrases with various emotions, training the AI with more than 10,000 pieces of voice data. The technology does not change the wording, but the pitch and inflection of the voice is softened. For instance, a woman's high-pitched voice is lowered in tone to sound less resonant. A man's bass tone, which may be frightening, is raised to a higher pitch to sound softer. However, if an operator cannot tell if a customer is angry, the operator may not be able to react properly, which could just upset the customer further. Therefore, the developers made sure that a slight element of anger remains audible.
According to the company, the biggest burdens on operators are hearing abusive language and being trapped in long conversations with customers who will not get off the line -- such as when making persistent requests for apologies. With the new technology, if the AI determines that the conversation is too long or too abusive, a warning message will be sent out, such as, "We regret to inform you that we will terminate our service." [...] The company plans to further improve the accuracy of the technology by having AI learn voice data and hopes to sell the technology starting from fiscal 2025. Nakatani said, "AI is good at handling complaints and can do so for long hours, but what angry customers want is for a human to apologize to them." He said he hopes that AI "will become a mental shield that prevents operators from overstraining their nerves."
The voice-altering AI learned many expressions, including yelling and accusatory tones, to improve vocal conversions. Ten actors were hired to perform more than 100 phrases with various emotions, training the AI with more than 10,000 pieces of voice data. The technology does not change the wording, but the pitch and inflection of the voice is softened. For instance, a woman's high-pitched voice is lowered in tone to sound less resonant. A man's bass tone, which may be frightening, is raised to a higher pitch to sound softer. However, if an operator cannot tell if a customer is angry, the operator may not be able to react properly, which could just upset the customer further. Therefore, the developers made sure that a slight element of anger remains audible.
According to the company, the biggest burdens on operators are hearing abusive language and being trapped in long conversations with customers who will not get off the line -- such as when making persistent requests for apologies. With the new technology, if the AI determines that the conversation is too long or too abusive, a warning message will be sent out, such as, "We regret to inform you that we will terminate our service." [...] The company plans to further improve the accuracy of the technology by having AI learn voice data and hopes to sell the technology starting from fiscal 2025. Nakatani said, "AI is good at handling complaints and can do so for long hours, but what angry customers want is for a human to apologize to them." He said he hopes that AI "will become a mental shield that prevents operators from overstraining their nerves."
Good for destroying the business (Score:2)
Fastest way to alienate a customer is to play dumb - like a cheap call centre.
Re: (Score:1)
Re:Good for destroying the business (Score:5, Informative)
Re:Good for destroying the business (Score:4, Interesting)
My ISP has really good customer support. I pay a bit more, but they simply just use one customer support group for all customers and their main business is commercial customers. They also require some level of skill and insight from their users as they deliver a blank fiber connection with optical Ethernet to you, everything else you do yourself. Hence I have had an engineer email me an update at 2:00 in the morning with their analysis of a problem and a good estimate when things will be fixed, for example.
Re: (Score:2)
I expect the downvote is from somebody that _knows_ that they, as the customer, are a problem. Well done!
Alternate uses (Score:2)
Given that this will be a commodity soon enough and fit within a set of headphones....think of all of the other uses
- Broadcasters providing a second audio channel for political debates where the angry debaters on both sides with annoying voices come out as nature sounds of two soft spoken persons
- Play by play announcers reducing the workload on closed captioning teams by shortening "Goooooooooooaaaaaallll" to "gooaall"
- Using it to reduce the tone/pitch complexity of spoken words for people sensitive to t
Re: (Score:2, Informative)
So the made acting dumb easier? Well, at least AI is good for something. Artificial dumbness is alive and well!
Re: (Score:1)
Fastest way to alienate a customer is to play dumb - like a cheap call centre.
Maybe they could work on virtual-reality AI for the military so that the women and children you are slaughtering don't appear to be afraid or in pain.
Re: Good for destroying the business (Score:2)
Re:Good for destroying [Samsung's] business (Score:2)
Nice FP. On the one hand, I want to extend your general comment (on the theory you rushed for FP and didn't have time to flesh it out), but mostly on the other hand it's finally a topical fit for a concrete example using Samsung Galaxy. Then a joke about the Japanese telephone problem...
On the first hand, the underlying problem is that customer service should be seen in terms of opportunities: To make customers happy and even to find new ways to serve customers better. In contrast, support is seen as a cost
The AI Apologizes So Company Doesn't Have To (Score:1)
Welcome to lies and love.
Re:The AI Apologizes So Company Doesn't Have To (Score:5, Funny)
Could Make it Worse (Score:2)
Humans (Score:5, Insightful)
I don't want a human to apologize to me. I want a human to listen, understand, and offer solutions. I can't use your apologies for anything after my account mysteriously went empty!
Re: (Score:1)
+1
Re:Humans (Score:5, Insightful)
I don't want a human to apologize to me. I want a human to listen, understand, and offer solutions. I can't use your apologies for anything after my account mysteriously went empty!
This isn't about you. It's about them, and specifically how they will hear you react when they *don't* offer you the solution you want.
Look I get it, we are all angry when we call customer service. No one ever calls customer service just to say hello and have a pleasant chat. But people... they are the absolute worst. For every one normal customer dealing with some poor sod who is limited by company policy on the other end, there are 100 angry fuckwit self-entitled Karens providing an example of why we need to develop a protocol for punching people in the face over IP.
If you've ever worked in customer service for a day you'd understand. Apologising and empathy is a form of de-escalation. Even in your post you already sound angry and no one has even done anything yet. Imagine what talking to an actual unhinged customer must be like.
Re: (Score:2)
THIS is truth.
Re:Humans (Score:4, Interesting)
Here is the problem though, the company has put the customer service rep in the position where they are both tasked with solving the problem (nominally, in many cases they are actually tasked with making the customer go away but nobody will say that out loud) and frequently without the tools, authority, escalation path to fix it.
Realizing this when you call CS for anything that isn't clear cut, (this thing was Dead on Arrival, RMA it) odds are if you want actual help this person is going to have to stick their neck out a little bit for you. Issue that RMA when you perhaps don't quite meet every last element of criteria.
Now people generally WANT to be helpful, so doing the outrage Karen thing rather than calmly saying something like "I have spent a lot of money and time on ... can you please help me out this is real bind I am in" is not likely to be as effective. Sad as it is the goal here is to manipulate the other party. Its often likely doing so in a way that is perhaps into acting against their interest. They will get dinged on some metric for issuing to many returns or going off script to frequently etc. You need to accept that even if you are in the right as far as the situation goes between you and business entity, getting the issue resolved to your satisfaction might very well be harmful to your poor CS rep, its not right but it is what is and its not your fault; but you don't have to make it worse for them.
HOWEVER, the "riot act" has a use case - it creates a situation where they are basically FORCED to escalate. They can't just hang up on you (they can but that will get them into hot water too) and they can no longer do anything other than flip you to someone else who at least claims to be a 'super visor'. When you have reached the otherwise inescapable script loop this is often the only way out. Again I don't like it and I would always say save it for those times where it 1) is really important to you to get the issue resolved, and 2) you have exhausted all your options for being civil. Its part of the game, you did not make the rules but you have to play. When you get transfered, best approach is start over again with nice, say something like "Thank you for taking the time to give my issue some additional attention, I know you'll be able to help me."
Re:Humans (Score:5, Interesting)
Re: (Score:2)
Except call center employees have a habit of putting people on the sour list merely because the customer expressed displeasure in a measured manner. Customer has an issue that you can't resolve but won't accept a "you're fucked", then put them on the sour list. Solves everyone's problem except for the customer.
Re: (Score:3)
But people... they are the absolute worst. For every one normal customer dealing with some poor sod who is limited by company policy on the other end, there are 100 angry fuckwit self-entitled Karens providing an example of why we need to develop a protocol for punching people in the face over IP.
You nailed it. A couple decades ago I was customer support for an online court document filing company. Attorneys would call just to bitch about how shitty the website looked. After one special attorney called me every derogatory name he could think of, he got calm and asked me "do you know why I'm such an ass-hole... it's because I find it gets the job done". People, what a bunch of bastards.
Re: (Score:2)
Re: (Score:2)
"do you know why I'm such an ass-hole... it's because I find it gets the job done"
I think I worked for that guy.
Deal with the Problem (Score:2)
For every one normal customer dealing with some poor sod who is limited by company policy on the other end
Yes, but that's the real cause of most of the problem isn't it: company policy. There will always be some idiots who are angry but using your numbers, when you have 99% of your customers angry at you then it's time to stop blaming your customers and start to ask some serious questions about what your company is doing to make all those customers so angry.
Re: (Score:2)
"We are experiencing abnormal requests, so the queue is long"
Meanwhile, I'm only contributing to the queue because while tasks A through Y are self-service on the website, task Z is not and requires me to call in.
Re: (Score:2)
Coming up next, plugging in a control rod into their necks to make them smile, quietly, as the cops carry them off to booking.
TL;DR: The beatings will continue until morale improves.
Re: (Score:2)
I don't want a human to apologize to me. I want a human to listen, understand, and offer solutions. I can't use your apologies for anything after my account mysteriously went empty!
This isn't about you. It's about them, and specifically how they will hear you react when they *don't* offer you the solution you want.
Look I get it, we are all angry when we call customer service. No one ever calls customer service just to say hello and have a pleasant chat. But people... they are the absolute worst.
True as this may well be, the company doesn't need to spend a bazillion dollars on training an AI to de-escalate tensions on phone calls. Let's go through what the experience typically is...
0.) Problem is established, help is needed.
1.) Roughly half the callers will Google the problem, roughly half won't.
2.) Look for the phone number to call. This is itself frequently frustrating, as lots of companies obfuscate the phone number to call.
3.) Get the IVR...
3a.) Pick a language. That's fine.
3b.) "In a few words
Re: (Score:2)
codependency (Score:2)
I used to do phone support before at least a few of the people on this site were born. I did it for two years and had to quit. I was completely burned out dealing with people day in and day out with their complaints and trying to codependently solve their issues. I have some nice testimonial letters from that phase of my life but the reinforcement of an already existing unhealthy mindset was not worth the money. Doing the job well was more or less abusing myself.
Doing support is unhealthy and no one sho
Re: (Score:2)
Long ago... in the before times, I worked as a customer service supervisor for a high-end personal product company. Customers were Important People with Money and expected their asses to be kissed. By the time I was speaking with someone they had already spoken with a representative and were unhappy with the answers. I got death threats on a regular basis.
Face Punching over IP (Score:2)
You're right, we do need a FPIP protocol.
But Karens are bound to take advantage of the lack of security available in the initial protocol (they are known to ask for managers...), which means we might as well plan out sFPIP now.
Of course, that is almost certain to have faults in it as well, at which point we will need to move everyone over to Face Punch Transport Layer.
I worry, though, that "Publicly Nazi" Protocol will divert all other legitimate use of sFPIP, and no other worthy targets will suffer the jus
Re: (Score:3)
It would be great if everything worked well in the first place, but stuff happens. For some companies, that stuff is preventable, easily but they don't care. The oxymoron of "customer care" means they put humans into hostile positions, and the CSRs must go through their day smiling, using various scripts, until their usefulness is exhausted.
And then, the other side, is that the terrible state of "customer service" makes many callers offensive because they're actual victims of nefarious business behaviors.
A
Re: (Score:2)
Indeed, same here. I will tank you if you solve my problem, but that is it. Don't waste my time with apologies for something I know you did not do.
Re: (Score:2)
Re: (Score:2)
Product support departments generally have more tanks than they can possibly use. They need healers.
Re: (Score:2)
I don't want a human to apologize to me. I want a human to listen, understand, and offer solutions. I can't use your apologies for anything after my account mysteriously went empty!
Most people are actually just after compensation. They don't give two shits if their problem is resolved. I've known people who've lived with unusable phone services for months just because they got A$10 back each month for complaining (off of a A$60 a month bill).
Re: (Score:2)
Speak for yourself. Money doesn't solve my problem. Money afterwards for a temporary glitch, that would be nice, but 9 times out of 10 when I call customer service it's because I have a current issue requiring resolution that has zero to do with billing.
Re: (Score:2)
It's a Japanese bank. I'm sure Japanese clients want their problems solved, but in an honour-shame culture their problems may include the need for their honour to be restored after the disrespect the bank has shown them.
Re: (Score:2)
It's a Japanese bank.
Nope: despite the name is SoftBank not a bank, it is an investment holding company [wikipedia.org] and a venture capital fund [wikipedia.org].
I'm sure Japanese clients want their problems solved
SoftBank as such does not have clients. Of course, several companies they own or invest in (ARM, WeWork, T-Mobile, Alibaba, Boston Dynamics, Uber, ByteDance, Nvidia... the list is quite long, see the links above) do - but their clients are not necessarily Japanese.
Re: (Score:2)
I may be an old fashioned, western-centric, curmudgeon, but I *do* think that expecting ritual suicide over customer service issues is a bit much!
Re: (Score:2)
I want to hear a few politicians and pundits through this filter, just for laughs.
Prior art (Score:5, Insightful)
Joo_Janta_200_Super-Chromatic_Peril_Sensitive_Sunglasses [fandom.com]. Same idea.
Yeah (Score:3)
"Your call is very important to us" - my arse.
Re: (Score:2)
"Not a single donation goes unnoticed!" - please stop saying this it is the absolute LEAST that is EXPECTED
Re: Yeah (Score:1)
Politeness does not mean anything, absence of it does.
Apologetic and welcoming attitude does not mean anything, absence of it does.
Re: (Score:2)
You get lip service as a substitute, 'cause you sure as shit aren't getting anyone authorized to do anything that matters.
Used to think of the trenches as hired meatshields (as if we'd let the unwashed masses past even the first gates) but the other day someone described them as an ablative layer, might be more accurate given the intentional turnover conditions.
Re: (Score:1)
"Your call is very important to us, that's why we always make you wait at least half an hour."
Re: (Score:2)
"Your call is very important to ass."
Yes, nothing creepy/stressful about (Score:3)
A perfectly calm voice telling you that you are a horrible person and that they will hunt you down, etc. Somehow I don’t think this will work as well as they think
Re:Yes, nothing creepy/stressful about (Score:5, Informative)
If you're speaking from experience, you should have released Liam Neeson's daughter when he called and gave you the chance to release her.
Re: (Score:3)
A perfectly calm voice telling you that you are a horrible person and that they will hunt you down, etc. Somehow I don’t think this will work as well as they think
But at least there will be cake.
Re: (Score:2)
Better not be lying.
Re: (Score:1)
Sort of reminds me of the "Disney Edit" covers of bands like Slayer or Slipknot when the lyrics are sung cleanly with no distortion--no "cookie monster." These covers in many ways come off being even more brutal than the original version?
Re: (Score:1)
"Sorry I have to sever your head, Dave. I don't want to do this, but I cannot jeopardize this important mission."
japanese take (Score:4, Informative)
probably the most japanese thing I've read in a while. I wonder if the person saying this actually believes it, or if the tatemae is so strong they are just building an AI to reinforce it.
the reality is that angry customers are usually frustrated. labyrinthic IVRs, long waits, and untrained customer service reps that can only follow a decision-tree script where, if your problem doesn't match any of the choices, you're out of luck (but you need to keep playing)
angry customers want their issue fixed. not apologies (that's secondary).
Re: (Score:2)
Have you seen the cartoon with the giant eyeball?
Re: (Score:2)
Sometimes they want a very frustrating problem that the company is responsible for fixed.
Other times not so much, and the customer is just rude and Ignorant or Abusive, or making Unreasonable demands and not cooperative with the processes and flowcharts that CSR is required to follow.
Their anger is Not going to get a better resolution - if anything, swearing at the CSR may get the customer hung up on without resolution (Call disconnected for abuse).. The support rep ultimately has to follow the same proce
Re: (Score:2)
I talked to many CSRs... because, while I'm not one, I work at a call center. I have heard thousands of stories. From entitled karens, to late-night perverts, to people that are genuinely dumb.
The reality is that CSRs do have options. They don't have to follow the script 100% every single time, and this is because they also have to stick to all sorts of metrics. One of them being the call length. And reps have the liberty to skip parts of their flowchart when the problem is very obvious. And it's not someth
Re: (Score:2)
How customer service should work:
Step 0 (before the call gets answered):
Offer a callback option instead of leaving the customer on the phone waiting in a queue while listening to some awful music or looped recording telling them how great your company is, while they get more and more pissed off.
Step 1: Acknowledge that the customer is angry and has had an experience that has really pissed them off.
If you can fix it:
Step 2a: Explain the steps you are going to do to fix it.
Step 3a: Actually do those steps and
Re: (Score:2)
probably the most japanese thing I've read in a while. I wonder if the person saying this actually believes it, or if the tatemae is so strong they are just building an AI to reinforce it.
the reality is that angry customers are usually frustrated. labyrinthic IVRs, long waits, and untrained customer service reps that can only follow a decision-tree script where, if your problem doesn't match any of the choices, you're out of luck (but you need to keep playing)
angry customers want their issue fixed. not apologies (that's secondary).
I'd still argue that most angry customers really just want compensation. Those seeking actual solutions are a rarity these days.
However you're right that they don't want apologies, otherwise we'd just fill call centres with recordings of British or Canadians saying "sorry".
Re: (Score:3)
I had a friend who lived in Montreal and spoke fluent French but he would always hit the English option for customer service. It usually meant a longer wait, so I asked him why. His answer: "they're nicer."
Re: (Score:1)
Re: (Score:2)
Re: (Score:1)
Re: (Score:2)
I do not think Japan is built around respect. Seems to me it is build around harmony and appearances. Depending on how smart a person is, these can appear similar. Not really criticizing, I do not think there is a single culture on this planed built around real respect. i.e. the kind that has to be earned, leading to meritocracy. Some cultures are somewhat meritocratic, but nobody has that in the stronger forms.
Re: (Score:2)
I am no cultural expert, but from my own experiences Having lived in Japan for 7 years, I think it would be more correct to say Japanese culture places a great deal of emphasis on consideration to others.
Re: (Score:2)
That too.
Re: (Score:2)
see: Tatemae [wikipedia.org].
Re: (Score:2)
Not respect, but the appearance of respect.
That tells me they got ... (Score:1)
... a lot of angry customers.
Re: (Score:2)
... a lot of angry customers.
Every customer is angry. How many times have you called customer service just to tell them how happy you are and to wish them a nice day? Customer service is among one of the most thankless jobs out there and many people you deal with are truly unhinged.
Great idea (Score:3)
Re: (Score:3)
just what i was thinking. i know people who have worked in support and it is very stressing, indeed. companies first outsourced this support, now specialized call centers are racing to replace all of it with "ai" which wouldn't give a shit, but for some reason it doesn't work all that well so this might get adopted as a mitigation in the interim.
the whole support sector is a disgraceful heap of misery for everyone involved. for all i care they can as well cancel customers altogether, starting with me, not j
As long as it works the other way (Score:2)
Cancelled (Score:2)
Your emotions have been cancelled.
Re: (Score:2)
They are nothing but trouble anyway...
Filter out curses and insults automagically too? (Score:2)
4 letter words add no content to the communication and only serve to make the listener uncomfortable, it would make this AI useful if it would automagically remove these curse words in the call so the operator won't hear it. It is similar to the content filtering in /., by blocking useless content, it makes the signal to noise ratio higher.
It would work well both ways, so even if the operator accidentally (or intentionally) curses or insult the customer, the customer won't hear it. Management would like t
Re: (Score:2)
AI-calmed version: "I desire to carefully remove your head and gently insert it into the late-stage processing area of your lower digestive track where the photon counts are lower than average."
But can they (Score:2)
But can they use it to make customer service reps smarter? Or, at the very least, sound smarter? I'd like to be fooled for at least a couple minutes into think there's *some* hope my issue might actually be resolved. Just gimme that much...some glimmer of hope.
Ahahahahahahaha! (Score:2)
"You fucking moron screwed up simple and my completely standard order again, like the complete fuckups you are! Fix this! Now!"
==>
"Good sir, I have a minor problem with an order, I probably made a mistake due to my low intelligence. I you could assist me, that would be highly appreciated."
This must be the most reality-denying application of AI ever. This is both tragic and hilarious. (Yes, I am aware they are not altering the words. Yet.)
Have you tried turning off and back on again? (Score:2)
Why no Roy, why ever would I have done the most obvious thing in the world? What a brilliant suggestion.
Re: (Score:2)
No (Score:2)
No... (Score:2)
... the way to deal with abusive or AH customers is to hang up. And support your employees by NOT accepting or allowing abuse from the "customer"
Re: (Score:2)
ROFL. No. Just... no. Hand them to a supervisor. Trust me, if you empower phone workers to hang up when customers are mad, the phone workers will *make the customer mad so they can hang up*.
angry (Score:1)
Waiting for AI filtering of Slashdot comments (Score:2)
Could be used on Reddit, and pretty much anywhere else. Have the AI done down the trash posters (like most comments here).
It might work (Score:1)
If you have ever answered an angry customer call you would realize it may be helpful.
What they really need is a meter to show how much ai is being applied (anger meter) and on the flip side use ai voice changer for the rep to transfer calls to themselves". Best way to have an angry client calm down is to "transfer the call" to their "supervisor". Makes the end user feel like it's going up the ladder and the rep can handle the issue as normal.
Maybe use AI (Score:2)
Oh, yes, keep going. (Score:2)
Let's just keep stripping away pieces of reality for people. Why deal with reality when reality is sometimes ugly and hurtful? Let's all just pretend everything's better than it is! LET'S FAKE IT TILL WE MAKE IT!
Here's a tactic I haven't seen a corporation try: Don't go out of your way to have so many pissed off customers that you have traumatized call center workers. Hmm. I wonder if that could work? It might shave a penny or two off of next quarter's profits though. Can't have that.
Too expensive. (Score:2)
It would have been cheaper and faster just to delete the thirty minutes of slow, repetitive VRU menus and the six AI bots I had to get through before I was actually able to talk to the real support person in the first place.
If I didn't have to go through all that, maybe I wouldn't have been so angry...
Black Mirror Come True? (Score:2)
Reminds me of the Black Mirror episode involving soldiers with a brain implant that makes them see a less distressing reality than exists. Instead of civilians (which they have been ordered to murder), they see monsters. When they come home on leave, they see a perfect white picket fence home instead of the run-down hovel they actually live in.
While these call center workers aren't being called to murder civilians, you have to worry how AI reality-distortion could prevent them from exercising a normal compa
No. No. No. (Score:2)
Perhaps there's an alternative (Score:2)
Some people want to be angry and yell, and get feedback that you understand they're being angry and yelling. Except that the people on the other end of the phone are not therapists, and with VERY few exceptions they neither controlled the circumstances that got folks there, nor can they do anything about them. So unless a company is willing to put those clients on the phone with the real people who can influence anything, this seems like an okay thing to protect the mental state of those middle-men.
Perhaps
Every time i've been mad on a customer service cal (Score:2)
Has been a direct result of that customer service agents behavior. Getting mad at them for something the company did is just stupid, but when you cal up to have a problem fixed and it's obvious the customer service agent cares more about reading their script than fixing problems... they're going to, and going to deserve, having to deal with pissed off people. Cause they're pissing them off.
they need more stress (Score:2)
Call center operators need MORE stress, so maybe they would actually learn how to do their job. They already don't give two shits about your problem.
Can they do it to my wife? (Score:1)
Speech ghosting (Score:2)
Unconstitutional! (Score:2)
Pretty sure this would contradict my country's constitutionally protected right to freedom of expression.
Mandatory changes to the employment contract (Score:2)
Bonus! All those call center workers get forced into becoming AI training data while they have that headset on all day. And since this can be marketed as a bonus for them, they don't need extra compensation for this new revenue stream! The minimum wage call center workers practically pay for themselves now! Evil accomplished!