MIT, Autodesk Develop AI That Can Figure Out Confusing Lego Instructions (theregister.com) 13
Researchers at Stanford University, MIT's Computer Science and Artificial Intelligence Lab, and the Autodesk AI Lab have collaborated to develop a novel learning-based framework that can interpret 2D instructions to build 3D objects. The Register reports: The Manual-to-Executable-Plan Network, or MEPNet, was tested on computer-generated Lego sets, real Lego set instructions and Minecraft-style voxel building plans, and the researchers said it outperformed existing methods across the board. Interpreting 2D instructions isn't easy for artificial intelligence. The researchers said there are a couple key problems going from visual instructions that, like Lego sets, consist entirely of images: Identifying correspondence between 2D and 3D objects, and dealing with a lot of basic pieces, like Lego.
Basic Lego bricks, the researchers said, are often assembled into complex forms before being added to the main body of the model. This "increases the difficulty for machines to interpret Lego manuals: it requires inferring 3D poses of unseen objects composed of seen primitives," the researchers said. Existing methods of parsing manual steps into machine-executable plans mainly consist of two forms, the researchers said: Search-based methods that are simple and accurate but computationally expensive; and learning-based models that are fast but aren't very good at handling unseen 3D shapes. MEPNet, the researchers said, combines both.
Starting with a 3D model of the components, the current state of the Lego set, and 2D manual images, MEPNet "predicts a set of 2D keypoints and masks for each component," the researchers wrote. Once that's done, the 2D keypoints "are back-projected to 3D by finding possible connections between the base shape and the new components." The combination "maintains the efficiency of learning-based models, and generalizes better to unseen 3D components," the team wrote. In the paper, the researchers said their aim is to create machines that help people assemble complex objects, and they include furniture alongside Lego bricks and voxel worlds in their list of applications. The researchers have made their code available on Github.
Basic Lego bricks, the researchers said, are often assembled into complex forms before being added to the main body of the model. This "increases the difficulty for machines to interpret Lego manuals: it requires inferring 3D poses of unseen objects composed of seen primitives," the researchers said. Existing methods of parsing manual steps into machine-executable plans mainly consist of two forms, the researchers said: Search-based methods that are simple and accurate but computationally expensive; and learning-based models that are fast but aren't very good at handling unseen 3D shapes. MEPNet, the researchers said, combines both.
Starting with a 3D model of the components, the current state of the Lego set, and 2D manual images, MEPNet "predicts a set of 2D keypoints and masks for each component," the researchers wrote. Once that's done, the 2D keypoints "are back-projected to 3D by finding possible connections between the base shape and the new components." The combination "maintains the efficiency of learning-based models, and generalizes better to unseen 3D components," the team wrote. In the paper, the researchers said their aim is to create machines that help people assemble complex objects, and they include furniture alongside Lego bricks and voxel worlds in their list of applications. The researchers have made their code available on Github.
But can it figure out... (Score:5, Funny)
... IKEA instructions?
Call me when that's in development. The entire planet will be grateful.
Re: (Score:3)
This is important. When we separate those who can read IKEA instructions from those who can't, we'll have a new Turing test for the upcoming AI apocalypse.
Re: (Score:2)
Oh hell no (Score:5, Funny)
and they include furniture alongside Lego bricks
In the Middle Ages they called them caltrops and were used to grievously wound enemies.
LEGO Instructions? (Score:1)
beginning of the end (Score:2)
Teaching robots to build more robots is the beginning of the end.
Re: (Score:1)
The only thing we cannot do better than another machine is build, another machine. What some of us have decided is that we need slaves for everything else or all the things we actually do better than slaves apparently but are to bored to do for ourselves. So we need slaves with machines to do....
My four year old (Score:3)
How is this news?
And people at LEGO... (Score:2)
What a happened to creativity? (Score:4, Insightful)
When I was a kid, Lego didn't have instructions or kits. You built stuff using your imagination. All these sets and kits do is encourage kids to follow the directions aka the rules. Damn commies.
Or.. (Score:2)
Or, they could just learn how to translate from Danish better than the monkeys who work for Lego.