Follow Slashdot stories on Twitter

 


Forgot your password?
Close
typodupeerror
×
Microsoft AI

Microsoft Speech Recognition Now As Accurate As Professional Transcribers (techcrunch.com) 176

Posted by EditorDavid from the speech-to-texts dept.
An anonymous reader quotes TechCrunch: Microsoft announced today that its conversational speech recognition system has reached a 5.1% error rate, its lowest so far. This surpasses the 5.9% error rate reached last year by a group of researchers from Microsoft Artificial Intelligence and Research and puts its accuracy on par with professional human transcribers who have advantages like the ability to listen to text several times. Both studies transcribed recordings from the Switchboard corpus, a collection of about 2,400 telephone conversations that have been used by researchers to test speech recognition systems since the early 1990s. The new study was performed by a group of researchers at Microsoft AI and Research with the goal of achieving the same level of accuracy as a group of human transcribers who were able to listen to what they were transcribing several times, access its conversational context and work with other transcribers.
This discussion has been archived. No new comments can be posted.

Microsoft Speech Recognition Now As Accurate As Professional Transcribers More

Microsoft Speech Recognition Now As Accurate As Professional Transcribers

Comments Filter:

  • Laughable Hype (Score:5, Interesting)

    by bwanagary ( 522899 ) on Monday August 21, 2017 @06:36AM (#55055851)
    On a daily basis in my work environment Microsoft technology is used to a) record voicemail and b) generate text from the speech.  Never, ever, have I received any converted voicemail that wasn't completely unintelligible gibberish.  Seriously.  This is utter nonsense.

    • Re:Laughable Hype (Score:5, Funny)

      by avandesande ( 143899 ) on Monday August 21, 2017 @07:37AM (#55056089) Journal
      You should start talking with people who don't speak gibberish.

    • Re: (Score:2)

      by skids ( 119237 )

      The missing part in this equation is the quality of the "human transcribers". I worked a few mturk transcription microjobs JOOC a decade or so back. Occasionally the job was to validate another person's transcription. It was rather awful. I don't blame them, though, because the pay is rather awful, too, especially for a job that pretty much monopolizes your attention.

    • I keep some of those emails, the transcriptions are hilarious.

  • Errors are not Errors (Score:5, Insightful)

    by idji ( 984038 ) on Monday August 21, 2017 @06:39AM (#55055861)
    When a human transcriptionist makes a mistake you can usually work out what they meant. When Speech-to-text (STT) makes a mistake it is often gibberish. So objectively it is "better" at transcribing, but subjectively much worse.

    • Re:Errors are not Errors (Score:5, Interesting)

      by AmiMoJo ( 196126 ) on Monday August 21, 2017 @07:17AM (#55056009) Homepage Journal

      Not any more. One of the ways that they got the accuracy up so high is by giving the machine an understanding of English and common phrases, similar to what a human has. It's been used for input correction on smartphones for a while too, e.g. with the Google keyboard it can correct the previous word based on the next one you type if it realizes that they don't make sense together.

      • Unless it actually understands what is being said then it will always make mistakes that result in gibberish

        If they are saying that they have cracked this, then they have strong AI, and should be announcing it to the worlds press ... (they haven't)

        They have added some syntax and grammar rules... just like everybody else ...

        • Re:Errors are not Errors (Score:5, Interesting)

          by AmiMoJo ( 196126 ) on Monday August 21, 2017 @08:03AM (#55056203) Homepage Journal

          It's more than just syntax and grammar rules. For example, Google has been mining the web for that kind of knowledge. You can see it in Google Translate sometimes. It generates suggestions for your input, and sometimes screws up like thinking "alot" is a word. It also uses colloquialisms in its output, which again it gathered from analysis of the web and which doesn't fit standard grammar or syntax rules.

          • Re: (Score:2)

            by Chaset ( 552418 )

            A few years ago, a colleague of mine and I were working in Japan. He was writing up a request for a quote and ran it through Google Translate to check his Japanese; expecting to get back an English phrase that at least vaguely corresponded to what he wanted to convey. All I remember was that the output contained the phrase "stormy bedroom". I had no idea how that came from his original text. Anyways, I told him to forget using Google translate.

          • Re: (Score:2)

            by arth1 ( 260657 )

            It's more than just syntax and grammar rules. For example, Google has been mining the web for that kind of knowledge. You can see it in Google Translate sometimes. It generates suggestions for your input, and sometimes screws up like thinking "alot" is a word. It also uses colloquialisms in its output, which again it gathered from analysis of the web and which doesn't fit standard grammar or syntax rules.

            Google Translate relies on community suggestions and validation. See https://translate.google.com/c... [google.com]
            The problem is that not everyone who joins there are truly fluent in both languages, nor all that literate.

            • Generally if you crowd source this you end up with a pretty good result. You don't need anywhere near "everyone" to make it work.

        • Re: (Score:3)

          by hord ( 5016115 )

          The way the machine learning databases are built, it does understand what is being said. That's why it is so effective. This happens through the connections that are built inside the neural network along with the architecture of the network itself. They are now using context-sensitive data labeling to assign specific meaning to words that are generally ambiguous based on the text around these words. The neural net can learn over time which combinations of words are likely to fall within specific categor

          • Re:Errors are not Errors (Score:4, Interesting)

            by djinn6 ( 1868030 ) on Monday August 21, 2017 @01:04PM (#55058017)

            The way the machine learning databases are built, it does understand what is being said.

            I think the word "understand" has a more general meaning than what you wrote later on. For it to understand what was being said, beyond making grammatical sense of the sentence, it needs to know the abstract concepts behind the words and be able to manipulate them.

            For example:

            Jeff is a software engineer, Kate is a software engineer, and Larry is also ...

            Can you finish the sentence?

            Most humans could do it with a high degree of accuracy. Some might even find the obvious answer so boring that they try for a more creative one. However, ML is still very far from that.

            Since it does not grasp the abstract concepts, its transcription is much more likely to lose meaning than a human transcriber. When talking about network technology for example, a human will not mis-transcribe "NAT" to "gnat", while a machine will.

    • Re:Errors are not Errors (Score:5, Informative)

      by jellomizer ( 103300 ) on Monday August 21, 2017 @07:22AM (#55056043)

      Normally we have transcriptionist who are trained in a particular area to understand the context of the message. A legal transcriptionist requires different training then a Medical Transcriptionist.

    • Just keep it recorded and have a human review it.

      This could cut costs greatly with this automation if it is true. Why pay 50 transcribers when you can pay 1 for a reduced wage since demand will now be lower and have the computer do the work for free?

      • Re:Errors are not Errors (Score:4, Interesting)

        by hord ( 5016115 ) <jhord@carbon.cc> on Monday August 21, 2017 @10:14AM (#55056887)

        I'm not a statistician but it's possible that once you can prove that the neural network can produce answers at a success rate higher than humans you would be introducing error by allowing humans to review it. I'm not saying it shouldn't be done but this is one of the weird questions that people will have to ask on a case-by-case basis as these technologies are applied to real problems.

  • Using it to post on slashdot (Score:5, Funny)

    by Harald Paulsen ( 621759 ) on Monday August 21, 2017 @06:49AM (#55055901) Homepage

    holyfield is these all of this was made worse by the fact that i had these birds skilled estimate uh... supplying itself what's your special prom to prevent fraud reform
    thoughtfulness julia roberts police comments entry drug connections predicting that nighttime beating

  • Comment removed (Score:5, Interesting)

    by account_deleted ( 4530225 ) on Monday August 21, 2017 @07:07AM (#55055955)
    Comment removed based on user account deletion

    • The recognition system is 5.9% accurate for the testers. For the rest of us it is far far off. Human testers are 5.9% accurate across a much larger selection of people.

      I can't use voice recognition to send a text without 3-5 attempts. And I don't have a hard accent.

      • Comment removed based on user account deletion

      • Re: (Score:3)

        by ranton ( 36917 )

        I can't use voice recognition to send a text without 3-5 attempts. And I don't have a hard accent.

        It is very odd that you have such a low success rate with voice recognition. At least 2/3 of my voice texts can be sent without editing, and most of the errors have to do with proper names. Are you sure you don't have an accent? My wife mumbles pretty bad when talking fast (so bad I don't like talking with her on the phone most of the time) but even she has a pretty easy job using voice to text now. It was pretty bad a few years ago but it really is amazing how much better it has become.

      • Re: (Score:2)

        by arth1 ( 260657 )

        I can't use voice recognition to send a text without 3-5 attempts. And I don't have a hard accent.

        I can't get voice controlled phone systems to work.
        The main problem is that I have a deep voice, and these systems are built on the pareto principle - cutting off the 20% with the deepest or highest voices is considered acceptable. I refuse to squeak to be understood.
        Some of the phone systems have hardcoded that if you say "human" or "operator", it will take you to a human operator. The problem is that it doesn't recognize those keywords either. After the aggressive high pass filter on the voice recognit

    • Re:Bad experiences on this front (Score:4, Interesting)

      by Baron_Yam ( 643147 ) on Monday August 21, 2017 @07:21AM (#55056037)

      5.9% means it still gets more than 1 in 20 things wrong. That's a LOT when you're feeding the information into a system that requires pretty much a 0% error rate.

      Second, there's a huge difference between standard language and specialist syntax. With programming, you're likely going to want a LOT of special formatting that you can type without thinking but it's cumbersome to communicate via speech in a way that won't confuse a speech recognition engine.

      And finally - so long as they don't have a related disability - a proficient typist can already type about as fast as they can form decent code in their head. With a bit of 'mousework' for selection and cut-and-paste I don't see speech ever becoming the superior entry method unless and until we have genuine AI that understands your intent rather than your words.

      It might be nice to use speech as a macro-invoker, though.

      • Comment removed based on user account deletion

      • With programming, you're likely going to want a LOT of special formatting that you can type without thinking but it's cumbersome to communicate via speech in a way that won't confuse a speech recognition engine.

        This story is about speech recognition being as good as transcription services. Programmers don't dictate their code verbally to be transcribed into text format by someone else, so that is a really weird thing to try to use as a counter argument.

        • >Programmers don't dictate their code verbally to be transcribed into text format by someone else, so that is a really weird thing to try to use as a counter argument

          Yet my post was in response to someone attempting to program by dictation, so somehow it seems completely relevant.

      • Speech is 4 or 5 times slower than typing.
        So unless you can tell an IDE "look at package 'my.product.model' and 'my.product.entities', create a Factory based on ctor signatures for all 'entities' that implement interfaces from 'models' and return 'model' classes" voice input is pretty pointless. And I doubt an 'AI' will be able to do that soon, while my template based code generator does that instantly. But I start it with a mouse click (which is slower than a key board short cut, obviously).

    • Writing code by voice? Are you insane.
      Speech is portraying ideas in a liner fashion. Coding you are jumping up and down filling different parts of the problem. At different time.

    • The reported error rate is for conversational English. This means that you cannot throw meaningless words at it. Modern speech recognition exploits grammatical and semantical structure. The stock recognizers can't do this for programming languages. You could train the model on a programming language, and certain constructs (like brackets, if-then-else) will see an improvement in recognition.

    • Do you have access to internal MS Research software? Cool, bro. Can you hook me up with some access too? Because you must've used the internal MS Research software to do your anecdotal testing some months ago, since you've got an opinion on how good it is at doing its job.

  • "As Accurate As Professional Transcribers" (Score:5, Funny)

    by Anonymous Coward on Monday August 21, 2017 @07:08AM (#55055967)

    "As Accurate As Professional Transcribers..."

    They left out "from Uzbekistan transcribing Navajo - underwater".

    Never trust anything Clippy say.

    • Re: (Score:2)

      by skids ( 119237 )

      They left out "from Uzbekistan transcribing Navajo - underwater".

      and "...working on cell phones with auto-correct enabled"

  • They should do tests using modern hardware. For example the speech recognition on iOS seems to be pretty good. If they can get this technology into windows 10 that would be awesome. Oh I dictated this using iOS.

    • You talk in a typewriter font ?

    • Re: (Score:2)

      by qbast ( 1265706 )
      I tried the same paragraph on iPad - flawless recognition. Then on Windows 10 - this is the result: "He shouldn't have been tested in what example dispute recall he can get distinctive into windows and that would be on a dichotic and"

  • The NSA would love this. Keyword scanning of 95% of what's spoken in phone conversations (given enough processing power to transcribe them all).

  • Yeah (Score:2)

    by Dunbal ( 464142 ) *
    Just make sure you run it on an air gapped computer if you want your conversation to remain private.

  • At work we have an cloud-based Outlook that transcribes voicemail to text. It's so comically inaccurate that we sometimes forward the results to the sender and we both get a good laugh.

  • Microsoft Speech Recognition Now As Accurate - Say (Score:3, Interesting)

    by WeBMartians ( 558189 ) on Monday August 21, 2017 @08:15AM (#55056259)
    If it can recognize "It's difficult to wreck a nice beach", I'll be thoroughly 'whelmed'.
  • IgPay AtinLay?

  • In which environment? (Score:3)

    by Opportunist ( 166417 ) on Monday August 21, 2017 @08:50AM (#55056421)

    In a sound proof studio built for sound recording spoken by someone with speech training?

    Or in an environment with 30 people talking in the background, an air condition running, doors and drawers slamming, people laughing, feet
    and chairs shuffling across the floor, some photocopiers that got their last service before Bush left office whining for hours and a person speaking into the phone while at the same time talking to coworkers and you're expected to know which words belong to you and which ones are directed at someone else?

    Aka "open plan office".

  • On the down side (Score:3)

    by fahrbot-bot ( 874524 ) on Monday August 21, 2017 @09:29AM (#55056621)

    It still showed up at the South Park "Save Films from their Directors" club for the wrong reason when it heard, "Free Hat" [wikipedia.org].

    (For those that aren't South Park followers...)

    Cartman writes "Free Hat" on the advertising poster in the belief that freebies are necessary to attract people. However, the crowd mistakenly thinks the rally is to free Hat McCullough, a convicted baby killer they believe was innocent.

    Now thinking that "Free Hat" would be a great name of one of those Windows App Store pirate streaming apps [slashdot.org] ...

  • Will it transcribe, "Diffused the situation," or "Defused the situation"? Every single TV closed-caption I've ever seen, and I've taken special note since I first became aware of this, has gone with the former. And those presumably have been humans making that error.
  • If you believe Microsoft without independent verification from an otherwise uninterested third-party who has no investment in the outcome, then you're a fool.

  • 5% (Score:3)

    by MMC Monster ( 602931 ) on Monday August 21, 2017 @11:43AM (#55057467)

    One in 20 words is wrong?

    How can a human transcriptionist be that bad?

    • Re: (Score:2)

      by gweihir ( 88907 )

      It is not. Sure, humans get a word wrong, but they will only very rarely mangle the meaning. Machine transcription, on the other hand, will often get meaning wrong and that is a serious problem.

      The only thing this shows is use of an unsuitable (in fact, utterly stupid) metric for marketing purposes.

  • Humans transcribers "have the advantage to be able to listen to the recording several times"? What utterly demented nonsense is that? Of course, the software, having the recording, can "listen" to it as often as it wants. There is absolutely no "advantage" here for the human transcribers.

  • puts its accuracy on par with professional human transcribers who have advantages like the ability to listen to text several times

    As if the audio sails by the program and isn't stored in memory and parsed as many times as needed.

  • I fuse micro sot noise recognition ball the time it words fall Leslie.

  • It is? And who decided *that*?

    We've got it on our hybrid phones. At least half the time, the voice transcription "preview" resembles, randomly, Vogon poetry, or perhaps only "computer poetry" from 40 years ago. It rarely gets a name or title correct, and the message they're trying to leave, *maybe* 50% is close enough to guess what they meant, without listening to the mp3.

Slashdot Top Deals

Don't tell me how hard you work. Tell me how much you get done. -- James J. Ling

Close