A Disturbing, Viral Twitter Thread Reveals How AI-Powered Insurance Can Go Wrong (vox.com) 49
An anonymous reader quotes a report from Vox: Lemonade, the fast-growing, machine learning-powered insurance app, put out a real lemon of a Twitter thread on Monday with a proud declaration that its AI analyzes videos of customers when determining if their claims are fraudulent. The company has been trying to explain itself and its business model -- and fend off serious accusations of bias, discrimination, and general creepiness -- ever since. [...] Over a series of seven tweets, Lemonade claimed that it gathers more than 1,600 "data points" about its users -- "100X more data than traditional insurance carriers," the company claimed. The thread didn't say what those data points are or how and when they're collected, simply that they produce "nuanced profiles" and "remarkably predictive insights" which help Lemonade determine, in apparently granular detail, its customers' "level of risk." Lemonade then provided an example of how its AI "carefully analyzes" videos that it asks customers making claims to send in "for signs of fraud," including "non-verbal cues." Traditional insurers are unable to use video this way, Lemonade said, crediting its AI for helping it improve its loss ratios: that is, taking in more in premiums than it had to pay out in claims. Lemonade used to pay out a lot more than it took in, which the company said was "friggin terrible." Now, the thread said, it takes in more than it pays out.
The Twitter thread made the rounds to a horrified and growing audience, drawing the requisite comparisons to the dystopian tech television series Black Mirror and prompting people to ask if their claims would be denied because of the color of their skin, or if Lemonade's claims bot, "AI Jim," decided that they looked like they were lying. What, many wondered, did Lemonade mean by "non-verbal cues?" Threats to cancel policies (and screenshot evidence from people who did cancel) mounted. By Wednesday, the company walked back its claims, deleting the thread and replacing it with a new Twitter thread and blog post. You know you've really messed up when your company's apology Twitter thread includes the word "phrenology." "The Twitter thread was poorly worded, and as you note, it alarmed people on Twitter and sparked a debate spreading falsehoods," a spokesperson for Lemonade told Recode. "Our users aren't treated differently based on their appearance, disability, or any other personal characteristic, and AI has not been and will not be used to auto-reject claims."
The company also maintains that it doesn't profit from denying claims and that it takes a flat fee from customer premiums and uses the rest to pay claims. Anything left over goes to charity (the company says it donated $1.13 million in 2020). But this model assumes that the customer is paying more in premiums than what they're asking for in claims. So, what's really going on here? According to Lemonade, the claim videos customers have to send are merely to let them explain their claims in their own words, and the "non-verbal cues" are facial recognition technology used to make sure one person isn't making claims under multiple identities. Any potential fraud, the company says, is flagged for a human to review and make the decision to accept or deny the claim. AI Jim doesn't deny claims. The blog post also didn't address -- nor did the company answer Recode's questions about -- how Lemonade's AI and its many data points are used in other parts of the insurance process, like determining premiums or if someone is too risky to insure at all.
The Twitter thread made the rounds to a horrified and growing audience, drawing the requisite comparisons to the dystopian tech television series Black Mirror and prompting people to ask if their claims would be denied because of the color of their skin, or if Lemonade's claims bot, "AI Jim," decided that they looked like they were lying. What, many wondered, did Lemonade mean by "non-verbal cues?" Threats to cancel policies (and screenshot evidence from people who did cancel) mounted. By Wednesday, the company walked back its claims, deleting the thread and replacing it with a new Twitter thread and blog post. You know you've really messed up when your company's apology Twitter thread includes the word "phrenology." "The Twitter thread was poorly worded, and as you note, it alarmed people on Twitter and sparked a debate spreading falsehoods," a spokesperson for Lemonade told Recode. "Our users aren't treated differently based on their appearance, disability, or any other personal characteristic, and AI has not been and will not be used to auto-reject claims."
The company also maintains that it doesn't profit from denying claims and that it takes a flat fee from customer premiums and uses the rest to pay claims. Anything left over goes to charity (the company says it donated $1.13 million in 2020). But this model assumes that the customer is paying more in premiums than what they're asking for in claims. So, what's really going on here? According to Lemonade, the claim videos customers have to send are merely to let them explain their claims in their own words, and the "non-verbal cues" are facial recognition technology used to make sure one person isn't making claims under multiple identities. Any potential fraud, the company says, is flagged for a human to review and make the decision to accept or deny the claim. AI Jim doesn't deny claims. The blog post also didn't address -- nor did the company answer Recode's questions about -- how Lemonade's AI and its many data points are used in other parts of the insurance process, like determining premiums or if someone is too risky to insure at all.
Re: (Score:2)
Right!
Insurance! Betting against yourself is a wise move.
Re: (Score:2)
Re:Typical (Score:5, Insightful)
Idiot customers are too stupid to understand how insurance works, get angry about it. Not much there, really.
Actually there's a few things going on here.
First, the insurance company bragged about how it denied claims for signs of fraud from AI analyzed videos... which sounds a bit like denying claims based on snake oil*.
Second, they bragged about their awesome loss ratio, which is good for investors, but to a customer that basically means you're charging too high of premiums.
Finally, they defended themselves against claims of AI-driven discrimination by almost perfectly describing a case of AI-driven discrimination. Basically, if your ethnicity correlates with higher risk their AI will reverse engineer your ethnicity, decide you're higher risk, and they think it's not discrimination. This is actually a fairly tricky technical problem to solve.
* Their subsequent explanation of facial recognition looking for claims via fake identities does sound more legit.
Re: (Score:3)
Finally, they defended themselves against claims of AI-driven discrimination by almost perfectly describing a case of AI-driven discrimination. Basically, if your ethnicity correlates with higher risk their AI will reverse engineer your ethnicity, decide you're higher risk, and they think it's not discrimination. This is actually a fairly tricky technical problem to solve.
It's an impossible problem to solve.
Different groups will have different risks. If you assess risk fairly, you will inevitably do some group sorting.
This is true everywhere, e.g. higher education, the composition of basketball teams ... only unfair, thumb on the scale policies can result in perfectly "diverse" sorting.
Hence obfuscation and double talk is inevitable. Well, or realism, I suppose - in theory. But you know which one we actually get in real life.
Re: (Score:3)
Depends on what you mean by fairly. If treating low-risk members of a group as if they posed the same risk as the average of the group is fair in your mind, then you'd be right, in your mind. But most people wouldn't think that'd be treating the individual fairly, at least not if they're the individual in question.
Re: (Score:2)
Depends on what you mean by fairly. If treating low-risk members of a group as if they posed the same risk as the average of the group is fair in your mind, then you'd be right, in your mind. But most people wouldn't think that'd be treating the individual fairly, at least not if they're the individual in question.
Until we invent crystal balls, groupings and average risks are the only things that insurance companies can use. You have to use statistics.
Actuary: "a person who compiles and analyzes statistics and uses them to calculate insurance risks and premiums."
Re: (Score:2)
Alternatively, they use the AI to know whom to assign their insurance investigators to. This cuts down on the amount of people any given insurance investigator has to investigate, thereby allowing them more time to prove any given case of fraud.
Demographics does have a real role in risk (Score:2)
When I was in my 20s, my insurance rate was higher than today even though I had a nearly flawless record because young male drivers tend to be higher risks than middle age male drivers. It was also highly deserved; even though my record LOOKED good, it was because I was a very good driver and I was riskier than I looked on paper like most young men.
Now add in additional factors like the higher level of impulsive behavior noted among black and Hispanic
Re: (Score:2)
discrimination /dskrimnSH()n/
the unjust or prejudicial treatment of different categories of people or things.
Note the key point, "Unjust". if a group is actually a higher risk, that treatment isn't discrimination because they ARE a higher risk.
How complicated is insurance, really? (Score:3, Informative)
Can't you at least come up with a better Subject in your desperation to first post? An interesting idea might be nice, too.
Anyway, it's not like I have an axe to grind. More like a bundle of axes to juggle, but the axe for this story is tagged TIDI, for Time-Inverted Disaster Insurance. It certainly could be "powered" by AI, but that's a kind of abusive verbing anyway. Second axe for today would be Covid-19, and I suppose there's a math axe in there, too.
But first I guess I need a bit of an ontology for bac
This is what really bugs me though (Score:3)
which help Lemonade determine, in apparently granular detail, its customers' "level of risk."
So the detail is large enough to see the grains, or small enough that it's grain-sized, i.e., coarse-grained or fine-grained? That's the part that really grinds my gears.
100% Lying BS for an answer (Score:5, Insightful)
Re: 100% Lying BS for an answer (Score:4, Informative)
"Anything left over goes to charity"
Literally the next sentence in the summary after the one that made you angry.
Re: (Score:3)
"Anything left over goes to charity"
That's BS. Charity funds are fungible. If you want to donate to a charity, donate to a specific one you know and trust.
There are too many charities, with questionable records, led by the girlfriends or wives or relatives of CEOs. Then the charity's funds just become a slush fund for the CEO's every whim.
Re: 100% Lying BS for an answer (Score:4, Informative)
"Anything left over goes to charity"
That's BS.
It's not. Lemonade is a registered Public Benefit Corporation, so they are legally obligated to operate in a way that's somewhat akin to a charity, even though they're still a for-profit company. They keep a flat 25% on any premium payment and use the other 75% to cover claims/reinsurance. Anything left from that 75% gets donated to charity once a year.
Read about their Giveback program [lemonade.com] for yourself. It's the key thing that differentiates them from other insurance companies, and the fact that it aligns their incentives with my own is why I actually switched to them about a year ago. I just had to make my first claim last month (for a brand new roof after a hailstorm), and the process couldn't have been simpler. They're half the cost of my previous insurance company and were WAY easier to deal with than the previous insurer was when someone ran a car into my house.
Re: (Score:2)
Left over (after the CEO bonus)
Re: 100% Lying BS for an answer (Score:5, Interesting)
Left over (after the CEO bonus)
Not so. The funds are pulled from a separate pool.
You can read up on Lemonade's business model and Giveback program [lemonade.com] here, but the gist of it is that they aren't a typical insurance company. Rather, they are a registered Public Benefit Corporation, and as such are legally obligated to operate for a social benefit. A flat 25% of its customers' premium payments go to their own costs, which is where they'd pay out any bonuses from, while the other 75% of the payments are used to pay out claims and purchase their own reinsurance. Anything left of that 75% at the end of the year gets paid out to charities that their customers select. So whether they pay out a claim or not doesn't matter: they aren't keeping that money, nor will it be available to pay their CEO a bonus.
Read up for yourself. Here's where they announced the switch to being a Public Benefit Corporation: https://www.lemonade.com/blog/... [lemonade.com]
Here's their FAQ: https://www.lemonade.com/faq [lemonade.com]
I've been using Lemonade for the last year or so and love them. They're half the cost of my previous homeowner's insurance, and I had no hassle at all when I made a claim a few weeks ago for a full roof replacement. The money to pay the roofers was even in my bank account before I saw the notification for the email where they told me they approved my claim.
Re:100% Lying BS for an answer (Score:4, Interesting)
I've worked for a major insurance carrier programming rates and what they are claiming is 100% bull! [...] And where does this money go if they don't pay for a claim? Oh I forgot. Denied claims are paid out in bonuses to the CEO and those who do the best at denying claims in their yearly bonus.
No, Lemonade doesn't use denied claims that way. Lemonade isn't a typical for-profit company: they're a registered Public Benefit Corporation [wikipedia.org] with a B-Corp certification, and as such are legally obligated to operate in a way that's somewhat more akin to what we might expect of a registered charity, even though they are still a for-profit corporation.
The way they do that is by "giving back" [lemonade.com] (check the Giveback section at the end) any premium payments they collect that aren't spent during the year. And to keep themselves from having an incentive to deny claims, the way that works is that they take a flat 25% from each premium payment (this is where bonuses would come from), then use the other 75% to cover the costs of paying out claims or purchasing their own reinsurance (i.e. insurance with Lloyd's of London for if they don't have enough to pay out claims). If there's anything left of that 75% at the end of the year, it gets paid out to charities selected by their customers.
So, whether it's paid out in a claim or paid out to charity, Lemonade never keeps any of the money related to denied claims. They keep the flat 25%, but claims aren't paid out of that portion, so they really don't have much reason to deny a claim, and if you check the reviews online, you'll find that the claims processes and approvals are simple and easy for the vast majority of their customers.
Including me.
I actually switched to Lemonade about a year ago after doing some research and loving that they had a business model that didn't incentivize denying claims. They were less than half the price of my previous homeowner's + VPP insurance and the process for signing up couldn't have been simpler, since it was nearly entirely automated. I had to make my first claim a few weeks ago after a hailstorm came through. Roofers suggested I needed a whole new roof, so I hit up the Lemonade app, clicked the button to make a claim, answered some quick prompts in a chatbot, recorded a 20-ish second video of me basically saying, "we had a hailstorm and this roofer next to me says I need a new roof", they sent out an adjuster a few days later, and my claim was approved the next day to the tune of $14K, with the first payment hitting my bank account even before I saw the email that said the claim was approved.
Couldn't have been simpler.
As for this current stuff, I'm not pleased to hear any of it, but it sounds like they're using the AI to flag suspicious videos for review while letting deemed-valid ones through without a human review. I'm fine with that. It's the converse with which I'd have an issue. Otherwise, I've frankly been so impressed by their operation and how smartly they've automated things to keep their costs low (while still making humans available when you need them) that I honestly just don't care about a Twitter tempest in a teapot or people on Slashdot who spout off without knowing relevant details about how the company operates.
Re: (Score:2)
You stupid shill. Stop posting advertisements for a shitty insurance company
If you’re gonna engage in ad hominem, you could at least make it interesting. We deserve better. You’re aiming at (and missing) low-hanging fruit if the best you’ve got is calling me a shill. I’m not giving up on you yet, but the fact that nearly your entire first page of comments has been downmodded to -1 suggests you haven’t yet figured out how to be witty or funny in your attempts at scathing critique.
Keep at it! Practice makes perfect.
Also, Lemonade has been a great insuran
Re: (Score:2)
Four people take out an insurance, then all four have an accident.
For their business model to work, they simply have to deny one person regardless of validity of their claim.
And that's at "stuffing money under mattress" payout level - dollar in, dollar out, disregarding inflation.
I.e. For them to be both making money AND giving to charity they'd have to be either a) denying claims like crazy, b) making money in other ways (i.e. lying about flat 25% off the top source of income), c) charging their customers
Re: 100% Lying BS for an answer (Score:1)
Re: (Score:2)
Such companies don't make claims that they're only taking 25% "off the top" flat fee, being a "public-benefit corporation" or that they are "keep[ing] themselves from having an incentive to deny claims incentive to deny claims" - while actively investing in algorithmic black boxes which were clearly trained to maximize "taking in more in premiums than it had to pay out in claims".
I.e. They are not practicing this particular form of bullshitting their customers - a psychological, feel-good, scheme for gullib
Re: 100% Lying BS for an answer (Score:1)
Re: (Score:3)
Four people take out an insurance, then all four have an accident [...]
Where to even start on how wrong your analysis is...?
- You're treating outliers as the norm
- You overlooked that premiums are sized to the risk involved
- You're ignoring the role of geography and scale in mitigating risk
- You've disregarded the reinsurance that has already been talked about in this thread
- You're acting as if you didn't just hear me say that they were half the price of my previous insurer
- You clearly missed them saying this is facial recognition tech being used to flag fraud based on ident
Re: (Score:2)
the way that works is that they take a flat 25% from each premium payment (this is where bonuses would come from), then use the other 75% to cover the costs of paying out claims or purchasing their own reinsurance (i.e. insurance with Lloyd's of London for if they don't have enough to pay out claims). If there's anything left of that 75% at the end of the year, it gets paid out to charities selected by their customers.
Interesting. My concern would be that they are incentivized to overcharge customers on the following basis: Their operating budget (salaries, offices, perks, etc.) is fixed at 25% of premiums charged, ergo the greater the premiums they charge, the more operating budget they have to pay themselves with.
If they consistently have money left from the 75% after paying out claims, then that would serve as evidence they are overcharging. This is something that would have to be analyzed on an ongoing basis -ther
Re: (Score:2)
Completely agreed. While all insurance companies are incentivized to increase prices so that they can pocket more, the fact that Lemonade only pockets 25% for itself means that if they want to bring in $X more, they need to raise cumulative rates by 4x that amount, whereas a traditional insurer would only need to raise cumulative rates by $X, since they get to pocket 100% of any excess.
For my part, they've been significantly cheaper than my previous insurance company, and 25% seems like a healthy amount to
Sounds like bullshit (Score:2)
That... does not match the common/garden meaning of the phrase "non-verbal cues". That sounds like something you come up with to distract attention while you're furiously back-pedalling because people are correct that you have an algorithm trying to play lie detector, and potentially picking up human biases from its training data in the process.
Excess Insurance Premiums (Score:4, Insightful)
The company also maintains that it doesn't profit from denying claims and that it takes a flat fee from customer premiums and uses the rest to pay claims. Anything left over goes to charity (the company says it donated $1.13 million in 2020).
Why isn't the excess insurance payments being returned to the customers?
I would think that lower premiums would bring in more customers.
It's a psychological, feel-good, scheme... (Score:2)
...for gullible middle-class property owners, particularly those believing in the fallacy of doing good by "voting with their dollars". [cnbc.com]
Last year, Oxfam found the world's 26 richest people were worth more than the poorest half of the global population combined.
Also, world's 2153 billionaires have more wealth than the 4.6 billion people who make up 60 percent of the planet's population. [oxfam.org]
Voting with dollars sure seems to be working out for people holding most of the dollars.
I'll Stick With USAA (Score:1)
Non verbal cues (Score:5, Funny)
Decker: "20, 30 maybe, cross-referenced."
Re: (Score:2)
Lol (Score:3)
I was involved with a startup incubator thing once. Bunch of MBA types and some aspiring AI company founders. We were chatting in the pub one evening and one of the MBA types was going on about AI-powered insurance. His idea was that you'd have an app that would monitor your driving, and give you a rate that varied in realtime based on AI assessment of your background, location, accelerometer input, etc.
I sipped my beer, laughed, and commented that people would absolutely hate that, and at some point your AI would commit PR suicide for you.
Ha! I looked at these guys at one point .... (Score:5, Interesting)
They promised a cheaper, better way to do car insurance, etc. etc.
I'm always happy to find a lower rate so I researched them a bit.
A buddy of mine in the industry said to watch out for them, because essentially, they just put customer money in a savings "pot" that starts over annually. This might work fine as long as things remain "business as usual". But the real risk is when you get into big disaster scenarios. Say there's an earthquake or hurricane damage along a swath of cities with a lot of people insured with them. The "pot" is going to get drained instantly and they're going to default on paying the remainder of the claims.
Re:Ha! I looked at these guys at one point .... (Score:5, Informative)
A buddy of mine in the industry said to watch out for them, because essentially, they just put customer money in a savings "pot" that starts over annually. This might work fine as long as things remain "business as usual". But the real risk is when you get into big disaster scenarios. Say there's an earthquake or hurricane damage along a swath of cities with a lot of people insured with them. The "pot" is going to get drained instantly and they're going to default on paying the remainder of the claims.
This is FUD. Like any small insurance company that doesn’t want to go bankrupt within a decade, they’re reinsured (in their case, through Lloyd’s of London) to deal with exactly that sort of scenario.
The way they operate is simple:
- They keep a flat 25% of premium payments.
- The remaining 75% is used to pay out claims and to purchase reinsurance. Anything left over at the end of the year from this pot gets paid out to charities that their customers select.
Thus, if a major disaster hits, they have reinsurance to cover it, the same as every other decent insurance company. And if a major disaster doesn’t hit, they pay out the leftover claims money to charity. Because they don’t keep the funds either way, they have no incentive to deny valid claims, so if you look through reviews online, you’ll find loads of people saying they’re really easy to work with.
That’s certainly been my experience with them. I switched to them a year ago for homeowner’s and VPP. They’re half the cost of what I was paying, provide better coverage, and I had zero hassle when I filed my first claim with them, just last month for hail damage. They approved a full roof replacement a day after the adjuster did his inspection, and the up front part of the $14K claim was in my bank account even before I saw the email saying they had approved the claim.
I had my doubts, but I did my research, have probably saved over $1000 in premiums from what I would’ve paid otherwise at this point, and had a great claims experience, so at least in my anecdotal, firsthand experience, they check out.
Re: (Score:2)
This is FUD. Like any small insurance company that doesn’t want to go bankrupt within a decade, they’re reinsured (in their case, through Lloyd’s of London) to deal with exactly that sort of scenario.
The way they operate is simple: - They keep a flat 25% of premium payments. - The remaining 75% is used to pay out claims and to purchase reinsurance. Anything left over at the end of the year from this pot gets paid out to charities that their customers select.
My insurer pays out a dividend if there is anything leftover in that "pot" at the end of the year. Nice to get a discount.
Re: (Score:2)
My previous one did as well. Their premiums were also twice as expensive after a few years of jumps for no apparent reason and the dividend amounted to a few bucks a month on a premium that was approaching $200 each month, but still, it was nice.
Re: (Score:2)
Aren't insurance companies insured against things like that?
Re: Ha! I looked at these guys at one point .... (Score:1)
Their actuary failed them (Score:1)
I for one welcome our new Insurance AI overlords (Score:1)
Lemondade! (Score:2)
I for one welcome our new, summer beverage named AI overlords.