The raw performance of the OMAP4 wasn't the issue, BUT the fact that it's an EOL architecture no longer supported by TI is showing in the current software quality of Glass. Ever since Google deployed KitKat to Glass (which has not been deployed in production to ANY other OMAP4 device), Glass has been unreliable and suffered from wildly inconsistent battery life. XE19.1 was a big improvement, but it was still a significant backwards step from pre-KitKat Glass. Then Google went and fucked it up again with XE21 - Twice in one week I had Glass run out of battery in only 8 hours with effectively zero usage other than sitting on my head idle. (1-2 notifications/hour, no Navigation, etc.)
Even before KitKat, the OMAP4 was a woefully inefficient CPU due to its age. A Snapdragon 400 with half the cores disabled would provide a MUCH better experience - more efficient/capable GPU, more efficient video encoding/decoding engine (no burning your head when recording), more efficient CPU.
I haven't worn Glass in nearly two months now. It's in desperate need of a hardware refresh to improve power management and stability, but Intel is the LAST thing Glass needs. Intel's mobile SoCs are worse than even Cortex-A15 in terms of power efficiency, which is why you see a number of Intel-based tablets and settop boxes, but next to no Intel-based phones (there are about as many Intel-based phones as Exynos5-based phones, another SoC that's woefully unsuitable to phones due to power consumption.)