It's been done (Score 3, Interesting)

An episode of PBS's Scientific American Frontiers back in April of last year featured an MIT Media Lab student named Brian Clarkson who built this exact same thing himself. He wore it like a backpack with fisheye lens cameras on the front and back. One of the more interesting things he was able to capture and re-watch was the first time he met his then current girlfriend.

You can watch the episode online.

(The part featuring Clarkson is titled "Never Forget a Face")