EFF launches new AI progress measurement project

Peter Eckersley writes: There's a lot of real progress happening in the field of machine learning and artificial intelligence, and also a lot of hype. These technologies already have serious policy implications, and may have more in the future. But what's the ratio of hype to real progress? At EFF, we decided to find out.

Today we are launching a pilot project to measure the progress of AI research. It breaks the field into a taxonomy of subproblems like game playing, reading comprehension, computer vision, and asking neural networks to write computer programs, and tracks progress on metrics across these fields. We're hoping to get feedback and contributions from the machine learning community, with the aim of using this data to improve the conversations around the social implications, transparency, safety, and security of AI.