Please create an account to participate in the Slashdot moderation system

 



Forgot your password?
typodupeerror
AI

AI Training Algorithms Susceptible To Backdoors, Manipulation (bleepingcomputer.com) 64

An anonymous reader quote BleepingComputer: Three researchers from New York University (NYU) have published a paper this week describing a method that an attacker could use to poison deep learning-based artificial intelligence (AI) algorithms. Researchers based their attack on a common practice in the AI community where research teams and companies alike outsource AI training operations using on-demand Machine-Learning-as-a-Service (MLaaS) platforms. For example, Google allows researchers access to the Google Cloud Machine Learning Engine, which research teams can use to train AI systems using a simple API, using their own data sets, or one provided by Google (images, videos, scanned text, etc.). Microsoft provides similar services through Azure Batch AI Training, and Amazon, through its EC2 service.

The NYU research team says that deep learning algorithms are vast and complex enough to hide small equations that trigger a backdoor-like behavior. For example, attackers can embed certain triggers in a basic image recognition AI that interprets actions or signs in an unwanted way. In a proof-of-concept demo of their work, researchers trained an image recognition AI to misinterpret a Stop road sign as a speed limit indicator if objects like a Post-it, a bomb sticker, or flower sticker were placed on the Stop sign's surface. In practice, such attacks could be used to make facial recognition systems ignore burglars wearing a certain mask, or make AI-driven cars stop in the middle of highways and cause fatal crashes.

This discussion has been archived. No new comments can be posted.

AI Training Algorithms Susceptible To Backdoors, Manipulation

Comments Filter:
  • Or make Skynet ignore the "chosen few" who send it on a rampage...

  • by Hentes ( 2461350 ) on Sunday August 27, 2017 @10:14AM (#55093003)

    Image recognition was never secure to begin with. If your security relies only on a visible image, that can be copied by anybody. People can set up fake road signs or break into facial recog using a photo of the owner. Hacking into Google and installing backdoors in the trained models is overkill.

    • by jwjr ( 56765 )

      Image recognition will be an important component of allowing autonomous robotic systems to function correctly. Robots will be more useful if they can recognize some thing by how they look, rather than requiring us to tag everything of interest in the real world with some secure system of correct identification. So anything that subverts image recognition raises a concern for safe and correct operation, rather than more typical computer security concerns, such as improper access control or authorization.

      S

      • People would not poison AI because "F*$& Google", they would poison AI for the same reason we see all sorts of criminal activity. Personal Gain and Money! That means the priority is exactly opposite your odd prioritization. Odd because it does not match crimes in _any_ market of any society.

        In terms of AI, there are too many possibilities to contemplate in a /. post. A simple few: Union funded AI corruption to maintain income, worked by people who are interested in corrupting AI to keep a job. Whi

        • by jwjr ( 56765 )

          To be sure, some people might poison it for personal gain.
          A person who embarrassed Google or another company developing autonomous robots could stand to gain by shorting their stock.

          As to your list of reasons for criminal activity, have you heard of terrorism?

          Finally, think of still other cases like the Iranian centrifuges.

      • Why not a school to train AI's? Not some super set of digital images, but a place(s) where AI's go to learn using their software?
  • Stop Calling This AI (Score:3, Informative)

    by Joviex ( 976416 ) on Sunday August 27, 2017 @10:24AM (#55093047)
    This is no AI.

    This is a huge database of weights, which are easily manipualted to be spit out, deterministically, from a computer i.e. NOT AI.

    News at 11.
  • The basic idea is that you can train AIs to make absolute associations when a specific pattern is recognized. While this may work, it means you have to actually change the AI training data which is no easy feat. Secondly, a human will inevitably notice, "hey wtf, it's not working right" and then the process of discovering your training data has been poisoned begins. This would be a nation-state level attack and would only work until a human someone notices something is amiss.

    I'm not losing any sleep over

    • What if they notice something amiss only as they turn toward a brick wall at 60 mph? Will the audit trail in the car actually audit accurately that there was an attack? Will an automaker shut down all their cars until the problem is found? Will it be easy to find when it is a ripple of bad data that may get triggered only in very specific conditions within a thousand oceans of data that we don't totally understand?
      • What if they notice something amiss only as they turn toward a brick wall at 60 mph?

        Then the vehicle runs into a wall, duh.

        Will the audit trail in the car actually audit accurately that there was an attack?

        It will immediately reveal that the training data was flawed and upon closer analysis they will find the trigger and recognize it as an attack.

        Will an automaker shut down all their cars until the problem is found?

        Not unless they all start running into walls.

        Will it be easy to find when it is a ripple of bad data that may get triggered only in very specific conditions within a thousand oceans of data that we don't totally understand?

        Nothing about investigating is easy, that's why it's an investigation. Remember when the Tesla car slammed into the tractor trailer? Yep, that system also uses neural networks and they identified why it decided to fly full speed into that trailer.

        • No they identified that weighting the sensors towards cameras was inadequate, and weighted the data towards the radar sensor. It had nothing to do with the AI. It is my worry that they won't shut down until all cars start running into walls, and that society at large will be left exposed to a potentially deadly issue without being told about it. That is a huge concern about AI, that there won't be full disclosure from companies of where their life-threatening issues are as the learning gets more and more
  • If this research raises concern that outsourced training of AIs may include back doors, a committee of separately trained AIs that "vote" on identifying things ought to address this threat, unless somehow the same backdoor is inserted into all committee members' training, which could be guarded against.

    This would also help to identify any such back doors, which could be found in an investigation whenever a particular vote is not unanimous.

  • Take the road signs for example.

    1. Start with a system to identify where the sign is - I'm not sure how to do that, but video motion identification might help.
    2. Next, take the center quarter of the sign area, and identify pixel colors. If the sign is strongly biased toward reddish pixels, it can't be a speed limit sign. General bins would seem to be red-and-white (stop, yield), yellow-and-black (hazard signs), white-and-black (speed limit, directions), green-and-white (lane identification, mile markers),

    • What if the 'reddish pixel' sign is on a hanging store sign or on a billboard or standing sign with a stop sign on it? How does the AI rule those out without understanding what advertising is? Also, I recently came from a place where there are yellow and black speed limit signs in school zones.
      • by Ken_g6 ( 775014 )

        Don't get me wrong, AI still has a job to do. I'm just suggesting classical algorithms can help avoid some obvious mistakes, and also can alert developers when the AI is attempting to make an obvious mistake and might need retraining. (Or might need backdoors removed in this case.)

        Where I live, yellow-and-black speed limit signs are usually optional, suggested speeds. Figuring out what a sign means needs to be done at a higher level than figuring out what a sign is.

        As for advertising with road signs, may

        • But avoiding obvious mistakes might be interesting theoretical discussion but it's not going to make a commercially viable solution.
    • Road signs are easy enough to solve: Just add "machine-readable" versions, beacons, or maintain an accessible database of them.
  • make AI-driven cars stop in the middle of highways and cause fatal crashes.

    That will only work until human drivers are replaced by self-driving cars that don't tailgate those compromised cars.

  • Ya, I'm gonna name names.

    Where are you TensorFlow? There's work to be done. Enough said.
  • by Anonymous Coward

    Mind your own business, Mr. Spock, I'm sick of your half-breed interference, do you hear?

  • Here comes the exploits, and they're not even on the roads yet!

    Just like with so-called 'smartphones', more and more I hear just reinforces my desire to never, ever ride in, let alone own, a so-called 'self-driving car', and to tell people you're nuts to trust your life to one.

[Crash programs] fail because they are based on the theory that, with nine women pregnant, you can get a baby a month. -- Wernher von Braun

Working...