Comming to think of it, a plain old 2D display has the same issues.
The distance from viewer to display is fixed, yet the watched content changes from close-ups to wide panorama, so both convergence and focal point are in conflict with what the viewer sees. On top of that the camera FOV creates permanently blurry areas that can't be fixed by the viewer changing focus. Blue tint on the picture of supposedly far mointains lies about the real distance and the focal point of the viewer is, again, in conflict colour shift preceived by retina. Moving viewers head doesn't show the scene from a slightly different perspective, as it should. A film watched from an angle looks really awkward. Camera movement isn't backed up by the inner ear receptors and that may lead to motion sicknes. Depth usually isn't essential for the story telling, but colour isn't either, and picture (radio anyone?) and sound for that matter (books existed long before movies).
The panic on the Lumiere Brothers train film shows clearly that cinema is in opposition to the natural human capabilities and a mere century certainly didn't change much in that respect - evolution doesn't work that fast.
Think how weird a person wearing early stereophonic headphones looked to people not too long ago. Almost as weird as a person wearing stereoscopic googles looks to many of us today. Think of all the issues with stereophonc audio, compared to the real world experience - stereo audio is not even close to real, just like stereoscopic video. To make it slightly closet, the 3D covers of BluRay discs use shots from 8 angles to mimic 3D picture, just like 7+1 audio systems do to mimic 3D sound.
The stereoscopic 3D is no that different from other techmologies. It's not perfect, but what is?
What is now proved was once only imagin'd. -- William Blake