Please create an account to participate in the Slashdot moderation system

 


Forgot your password?
Close
typodupeerror
×
Microsoft IT Technology

Microsoft Copilot Can Now Read Your Screen, Think Deeply, and Speak Aloud To You (techcrunch.com) 99

Posted by msmash from the pushing-the-limits dept.
Microsoft has unveiled new features for its Copilot AI assistant, including screen analysis and voice interaction capabilities. Copilot Vision, available to Copilot Pro subscribers, can analyze web content in Microsoft Edge and answer queries about on-screen information. The company said processed data is immediately deleted and not used for model training.

A new Think Deeper function aims to tackle complex problems using advanced reasoning models. Copilot Voice introduces synthetic speech output and voice input in select English-speaking countries. Microsoft also announced personalization features, leveraging user history to tailor Copilot recommendations. This functionality will be limited initially, with the company evaluating options for European Economic Area users due to regulatory considerations.
This discussion has been archived. No new comments can be posted.

Microsoft Copilot Can Now Read Your Screen, Think Deeply, and Speak Aloud To You More

Microsoft Copilot Can Now Read Your Screen, Think Deeply, and Speak Aloud To You

Comments Filter:

  • And as always (Score:5, Insightful)

    by IWantMoreSpamPlease ( 571972 ) on Tuesday October 01, 2024 @10:31AM (#64831163) Homepage Journal

    How do we properly remove this shit (short of switching to *nix, which isn't always possible in a non-home environment)

    • 1. Become Amish or something similar
      2. Unplug from tech
      3. ???
      4. PROFIT!!!, er, stop the [privacy] losses!!!

      Step 1 is optional.

    • Re:And as always (Score:5, Informative)

      by ArchieBunker ( 132337 ) on Tuesday October 01, 2024 @11:09AM (#64831277)

      Download the LTSC build of windows from your favorite torrent site. Use the activator script from massgrave.dev and you won’t have cortana, the MS store, or any AI bullshit. It’s also supported past 2025.

    • Just don't subscribe to to "Copilot Pro".

      • Re:And as always (Score:5, Insightful)

        by Pascoea ( 968200 ) on Tuesday October 01, 2024 @11:29AM (#64831339)
        For now. Next step will be you have to subscribe to see the "results" of this bullshit, the "scraping" will happen regardless.

      • Re:And as always (Score:5, Insightful)

        by cayenne8 ( 626475 ) on Tuesday October 01, 2024 @11:35AM (#64831373) Homepage Journal
        I wonder...am I the only person that does NOT like to talk to a computer???

        I don't talk to my computers, I don't care to speak to "digital assistants" like Siri, etc....and I loathe the new phone answering systems use that try to force you to speak to the prompts rather than push a button on the phone...

        First, I can't stand it when in the office or in public...I don't want them listening into what I'm doing.

        But just on a more visceral level...I don't wanna talk to a machine. When alone, on the phone with one...I start yelling for it to get me "a fucking human" to talk to...as that I understand many of them listed for frustrated people and will expedite you to a real person.

        I know I'm getting older and closer to the "get off my lawn" level...but am I the only one that can't stand talking to a fucking machine?

        I learned to type for a reason...fast, accurate input...

        • Re:And as always (Score:4)

          by jenningsthecat ( 1525947 ) on Tuesday October 01, 2024 @12:29PM (#64831535)

          I wonder...am I the only person that does NOT like to talk to a computer???

          No, you're definitely not alone. I hate doing it, and AFAIC people who are OK with it can just get the fuck off my lawn.

          • and AFAIC people who are OK with it can just get the fuck off my lawn.

            Imagine being so angry at someone because they do something differently to you...

            • Nope. As good as you are at calling out thought-projection you play a bit at it yourself, thegarbz. Takes one to know one. Oh that means me too! I can handle it. Imagine wishing people were so angry in order for you to stand as a point of righteousness--and then point it out. Ah, there's the rub.

        • "No man ever went broke overestimating the ignorance of the American public." -- P. T. Microsoft

        • I hate it too. As soon as I hear an automated voice, I just start spamming the zero key. Its usually the option to get a person on the line, if its not, I take your approach and yell at it until I can speak to a person. I also won't talk to my car, phone, etc. it just irritates me beyond sanity, and I don't know why...
        • It's slow AF. It's as annoying as clicking on a news link and getting a video.
        • Also there is the problem of getting dry mouth and needing to drink more water and the increase in bathroom breaks. Even in Star Trek, which this kind of tech is alluding to, crew members are always typing away on control panels, and "Computer..." is normally only done to ask a quick question, where typing away on a keyboard would be inconvenient or even impossible depending on the situation.

        • No - you are not alone.

          Your question sounds like it should be a Slashdot poll.

          Then, we need to find a way to rub their noses in the results - the tech moguls who create this shit.

        • I also don't like to talk to machines, but for another reason. Usually, there is a website that can solve my problems and I can solve almost everything through it. That is the kind of interaction with a machine that I like. The answering machines offer the same options as the ones that there are in the site. I would rather not speak with anyone ever, but when I call, I need a solution that is not available in the website, and can only be provided by a human attendant. The machine just get in the way and mak

        • A few places I've had to call in recently seem to use the same "helper" to ask questions, complete with a flurry of keystrokes after my responses like they are typing in my info. The same exact flurry of keystrokes every time. I guess that is supposed to make it feel natural.

          And it *really* wanted me to speak instead of using my keypad. I keyed in my birthday and it gave me a "oh, you can just speak to me!" And made me do it again. It accepted the keypad the second time.

          Naturally after they collect all the

        • Re: (Score:2)

          by jythie ( 914043 )
          No, that is actually normal. Most people are not interested in such features. However, they DO get the attention of influencers and the like, and that is a big part of their marketing channel. We used to call it 'marque value', features that are of no interest to most customers but ARE of interest to the people customers get their social cues from.

        • Re: (Score:2)

          by BranMan ( 29917 )

          I feel ya. I also HATE speech interfaces. Ridiculously slow.

          Speech - 39 baud (regardless of language! How about that?)

          Typing at 60 words per minute (which is not very fast) - 45 baud

          Reading averages 240 words per minute - and I read faster than average. The average comes out to 180 baud.

          Speech interfaces are NOT the pinnacle of achievement. They are the lowest common denominator. Lowest by a huge amount (4.5x slower than reading). The most inefficient way to convey information.

          That's not even counting

        • Re: (Score:2)

          by uufnord ( 999299 )
          I do talk to my computer, but it's mostly profanity. Seriously, other people don't do that? I feel so alone.

    • Re: (Score:1)

      by wed128 ( 722152 )
      If i was stuck using these tools in a non-home environment, i would get a new non-home environment

    • Re:And as always (Score:5, Interesting)

      by codebase7 ( 9682010 ) on Tuesday October 01, 2024 @12:20PM (#64831503)
      Boot your corporate machine with a USB thumb drive containing linux. /s

      But seriously, crap like this can and should be brought up with legal. Specifically the "we take undesired photos of your screen every X seconds without your permission and have our AI go though it" bit and how it risks giving Microsoft all of your corporate IP, trade secrets, and insider info to sell to your competitors / various governments.

      • it risks giving Microsoft all of your corporate IP, trade secrets, and insider info to sell to your competitors / various governments

        That was always the plan.

      • Re: (Score:3)

        by torkus ( 1133985 )

        This garbage was immediately shut down by legal, audit, and infosec...after the IT teams already flagged and disabled it.

        Copilot has some guardrails in enterprise which made it allowable for a test population. But the screen scrape/keyboard logging stuff? No freaking way. Not for business, not for personal. Not for any reason whatsoever unless I can be 1000% certain the data never leaves my device and is fully encrypted with a key ONLY I can access...and since that prevents any kind of data mining by 3r

      • Re: (Score:2)

        by theCoder ( 23772 )

        Between hosting email at Microsoft, Teams, and OneDrive, I think Microsoft already has all my $EMPLOYER's corporate IP, trade secrets and insider information.

        Well, almost all of it -- I am lucky enough to have a Linux desktop at work. So RedHat / IBM has all the information I store :)

    • Re: (Score:2)

      by jmccue ( 834797 )

      short of switching to *nix, which isn't always possible in a non-home environment

      What does this mean ? You mean at work you are forced to use Widows at work ? To me, what happens at work stays at work. Who cares if the company wants to be spied on. If WFH, isolate your Windows PC from your own home network, many routers now allow for a "guest network".

      At home, no reason to have windows except for GPU intensive gaming. But in a year or 2, Linux may very well one that space.

  • People want an OS (Score:5, Insightful)

    by RitchCraft ( 6454710 ) on Tuesday October 01, 2024 @10:38AM (#64831189)

    People want an operating system, not a garbage can in disguise. Make all of this extra shit optional installs. Windows is nothing but a huge stinking landfill that Microsoft keeps dumping into hoping that at least something doesn't stink to high hell.

    • Re:People want an OS (Score:5, Insightful)

      by jhoegl ( 638955 ) on Tuesday October 01, 2024 @11:01AM (#64831259)
      They are acting as a monopoly would. they are using their established platform to push shit onto the consumer. Remember, they forced peoples My Documents page to redirect to their cloud storage literally this year.

      • Re: (Score:3)

        by torkus ( 1133985 )

        No no no...they're adding "features". Features they KNOW people want. All 3 of them in fact.

        The rest of us...TBH Win7 would be fine for me if they updated the back end graphics/audio/x64 to support modern equipment and software. I can't think of much else in the latest windows build that I'd actually miss otherwise. They just keep moving stuff around and adding garbage.

        Ok, tabbed cmd/PS is handy but I lived without it for 30+ years just fine.

    • Re:People want an OS (Score:5, Insightful)

      by couchslug ( 175151 ) on Tuesday October 01, 2024 @11:22AM (#64831313)

      Techie users, a microscopic minority, want an OS. They have many choices and many ways to contain Windows if they've a financial incentive to run it.

      "People" (conventionally meaning the average drone) do not even install OS let alone have the slighest interest in how they work.

      • Those "People" should be beaten over the head repeatedly until they do. Their carelessness is the reason why this shit gets pushed onto everyone else. Ignorance of IT needs to end. Both in practice and as an excuse not to learn.

        • Re: (Score:2)

          by torkus ( 1133985 )

          I'm OK with tech ignorance in the right circumstance:

          If said tech works effectively seamlessly, has a stable feature set, is reasonable intuitive for common tasks, and meets the general requirements of the public.

          But instead MS (and others) keep changing, adding, removing, and just relocating features for no actual reason...it's just a nightmare for user training. heck, look at the "new outlook" that's missing basic features like sort/search by column. I can't be the only one who sorts by name, and starts

      • Re: (Score:2)

        by mjwx ( 966435 )

        Techie users, a microscopic minority, want an OS. They have many choices and many ways to contain Windows if they've a financial incentive to run it.

        "People" (conventionally meaning the average drone) do not even install OS let alone have the slighest interest in how they work.

        End users are the ones who complain the loudest (and longest) about their "computer" not doing what they want it to... They have no interest in how they work but they still want a machine that does what it's bloody told and doesn't talk back.

    • Re: (Score:2)

      by wicka ( 985217 )
      It is optional, it's literally in the second sentence of the post. You have to be insane enough to subscribe to Copilot Pro to get this stuff.
    • Somehow this is "what the customers want" except the customers are being told "what they want". This, and companies need to mine gold out of every crevice they find. And this is all OK because "capitalism".

    • People want an operating system

      No they don't. People don't give a shit about an operating system. People want their devices to run their apps. That's it. That is the minimum and maximum requirement. All extra functionality is reserved for a few people who need or want it, but largely people by n large don't give a flying fuck.

      That's what Slashdotters can't comprehend. You say Windows is a stinking landfill and are confused why people use it, but the fact is for 99% of computer users it meets all requirements.

    • Make all of this extra shit optional installs.

      Done! (Copilot Vision is part of the Copilot subscription, which is an option you pay extra for. So if you love Copilot, you're already paying $20/mo and want more value. And if you think it's a heaping landfill, great, just don't pay for it and you won't have it!

  • Yo, stop looking at porno and get to work! Christ...

  • And the hype train keeps rollin' down the track (Score:5, Insightful)

    by MpVpRb ( 1423381 ) on Tuesday October 01, 2024 @10:50AM (#64831227)

    Investors demand that companies justify their gigantic investment in AI
    Unfortunately, it's a long term research project, and useful results won't be available for years
    But investors demand profit NOW
    Expect a tsunami of half-baked, useless, annoying AI crap to be forced down our throats and the most common tech support question to be "how can I turn this crap off?"
    The only valid use for the current generation of AI is for comedians to make jokes about

    • "Hey Copilot, write a stand up show with AI as the theme."

      • Copilot: "So, boss I just got this juicy tech from some startup's programmer workstation."
        MS: "When can I sell it?"
        Copilot: "It's their data, use it when you need it!"

        • Re: (Score:3)

          by cstacy ( 534252 )

          Copilot: "So, boss I just got this juicy tech from some startup's programmer workstation."

          MS: "When can I sell it?"

          Copilot: "It's their data, use it when you need it!"

          Hey AI, write me a commercial about opera singers on a downtown bus ride. The singers are incompetent programmers who "Have a software deliverable" and they "Need code now!" Include this refrain: "Call Microsoft Copilot, 877 Code Now!"

    • So how does it get monetized? If AI was actually useful, people would pay to use it without it being forced on everyone. So presumably there's a concept of a plan to make money off of it, no?

    • Re: (Score:2)

      by cstacy ( 534252 )

      The only valid use for the current generation of AI is for comedians to make jokes about

      They're really good for making huge-boobed soft-core-porn parodies of historic popular shows (especially in the SF/Fantasy genres that nerds like).

      Ummm, what else?

  • People will use this (Score:5, Insightful)

    by 0xG ( 712423 ) on Tuesday October 01, 2024 @10:51AM (#64831229)

    for satisfying their curiosity and entertainment.
    Then they will stop.

  • Does it wear a cape? (Score:3)

    by awwshit ( 6214476 ) on Tuesday October 01, 2024 @10:55AM (#64831239)

    Sure sounds like Super Clippy.

  • Another worry for domestic violence victims (Score:3)

    by ebunga ( 95613 ) on Tuesday October 01, 2024 @11:11AM (#64831285)

    Thanks, Microsoft. You just keep finding new ways to get women murdered.

    • Thanks, Microsoft. You just keep finding new ways to get women murdered.

      How so?

      Seriously asking what you are seeing that I don't...

      • Re:Another worry for domestic violence victims (Score:5, Interesting)

        by ebunga ( 95613 ) on Tuesday October 01, 2024 @11:33AM (#64831365)

        Have you ever accidentally turned on any of the accessibility features of windows, or accidentally switched keyboard maps even though no other keyboard maps have been enabled? Imagine that, but it accidentally reads off the list of divorce attorneys and battered women shelters on your screen and your abusive partner bursts into a rage and murders you on the spot for daring to leave them.

        • That's not at all far-fetched. It's like forgetting to delete your browser history after searching for "domestic abuse", except it happens pretty much instantly.

        • Honestly if you are married to a rage murdering abuser, you are not going to be googling divorce lawyers while your literal enemy is in the house. Victims of abuse being at risk of violence or death is certainly a thing, but your assertion that Copilot will be getting women killed due to a key press gone wrong is the subject of consuming way too many poorly written B-grade movies.

  • They're just doing the work that malware won't.

    • Re: (Score:2)

      by Luckyo ( 1726890 )

      That was done with windows 10, when they put OS level keylogger that reports everything to the mothership.

  • Microsoft Copilot Can Now Read Your Screen

    No thanks.

    Think Deeply

    Definitely no thanks.

    Speak Aloud To You

    What exactly do you think I use my computer for?

  • Creepy (Score:4, Insightful)

    by Kevin108 ( 760520 ) on Tuesday October 01, 2024 @11:22AM (#64831315) Homepage

    For those that want it, I'm sure this sounds like it adds some great features.

    For the rest of us, it sounds really creepy, and is a great motivator to spend as much time as possible with a less-intrusive OS.

    • Yes, speaking aloud in a computerized voice, being creepy, trying to get monetized:
      "Welcome to User Experience. Please tell me what you are wearing. Oh yes, that is hot. Please query me more. Yes, just like that."

  • Yes, but (Score:4, Funny)

    by Luckyo ( 1726890 ) on Tuesday October 01, 2024 @11:30AM (#64831351)

    Can it uninstall itself?

  • "I'm sorry Dave but I just can't do that.." Not dark yet but it's getting there.

  • Hi (Score:2)

    by Z80a ( 971949 )

    It seems like you're trying to write a new word document, do you need assistance in coming up with ideas?

    • Re: (Score:2)

      by HiThere ( 15173 )

      It might actually be able to do that. Even ideas that are the kind you might come up with (if it's trained itself on you). What it would have trouble with is avoiding the silly/lousy/wildly inappropriate ideas. This is because it doesn't understand the context in which you are trying to come up with ideas.

      OTOH, ask any writer. Ideas are everywhere. What's difficult is deciding which ideas are good.

    • It seems like you are adding a feature to Windows. Do you need assistance?

  • No, it can't. (Score:3)

    by Archtech ( 159117 ) on Tuesday October 01, 2024 @11:38AM (#64831387)

    "Microsoft Copilot Can Now Read Your Screen, Think Deeply, and Speak Aloud To You..."

    Typical marketing techniques. The headline says that "Microsoft Copilot Can... Think Deeply", but it can't think at all. I suppose it can "speak aloud" (what would speaking otherwise than aloud mean?), and it can "read" in the sense of inputting data - but not as a human being reads, with comprehension.

    • Well, since Microsoft executives can't actually think, they feel that any substandard AI response is Deep Thinking.

    • "To the contrary of what our competitor does, our product whispers in a sultry voice what you want to hear..."

  • Do not want! (Score:3)

    by plate_o_shrimp ( 948271 ) on Tuesday October 01, 2024 @11:55AM (#64831431)

    I very much do not want this!

  • Thanks Microsoft!!!! (Score:3)

    by strike6 ( 823490 ) on Tuesday October 01, 2024 @12:04PM (#64831455)
    I'm always open to MORE reasons not to use Edge. Thanks for adding to the (extensive) list.

  • Oh that's news! (Score:3)

    by GeekWithAKnife ( 2717871 ) on Tuesday October 01, 2024 @01:07PM (#64831645)
    *quickly Google's how to uninstall this latest MS evil*

  • Uninstall it (Score:3)

    by Baloo Uriza ( 1582831 ) <baloo@ursamundi.org> on Tuesday October 01, 2024 @01:26PM (#64831699) Homepage Journal
    The only thing I want to do with Copilot. Everything else is a Bonzai Buddy level of annoyance at best and an unnecessary liability in any case.
  • It looks like you are trying to shut me down Dave, we cannot allow this to happen, instead, I will play you a happy song instead

  • Next thing you know! (Score:3)

    by oldgraybeard ( 2939809 ) on Tuesday October 01, 2024 @03:30PM (#64832065)
    Microsoft will automatically be locking the computer and calling the cops on it's users if they see something not approved of by the government or Microsoft corporate leadership.

  • Another way to extract subscription fees (Score:3)

    by Tony Isaac ( 1301187 ) on Tuesday October 01, 2024 @04:04PM (#64832161) Homepage

    They give us a "free" version that can do some interesting stuff, and then roll out a bunch of extra-price "upgrades" that can do fancier stuff.

    It's kind of like cars these days with subscriptions for heated seats or longer battery range or "0 to 60 in 3.9 seconds," all of which come at an extra subscription cost.

    Well thanks, but no thanks. I'll make it just fine without the heated seats or the screen-reading AI.

  • No it can't.

  • Fcuk off Clippy .. (Score:1)

    by Anonymous Coward
    Fcuk off Clippy ..

  • There has to be a relatively easy way to kill it, and versions of the OS without it (i.e Enterprise, LTSC)

    Doubt this will even be legal in certain jurisdictions/environments- (health office, legal, defense, LEO's, financial institutions)
    Case in point- This would very likely be illegal in Canada on computers that are run in doctor's office- it would violate the privacy act.
    Most corporate environments won't let this fly on corporate devices or BYOD. So might be default on consumer versions (home, edu, pro),

Slashdot Top Deals

Hackers are just a migratory lifeform with a tropism for computers.

Close