I'm not really convinced that too much functionality would hurt sales - but it's a fact that the lackluster functionality of Nspire calculators, prior to Ndless, _did_ hurt sales ;-)
It's pretty easy to understand why: Nspires are expensive, and users don't get much for their money. The first version of the OS didn't even contain any form of BASIC programming (!!), and that four years later, the BASIC of Nspires remains sub-par compared to the TI-Z80 and TI-68k BASIC.
As for TI calculators being banned from education... people deciding such a blunder would be _severely_ incompetent:
* an unfixable exploit which enables hack-ish installation of arbitrarily modified OS on TI-68k calculators has been known for 11 years... but it did not get TI calculators banned from standardized tests for that very reason.
* the factorization of all interesting 512-bit RSA public keys used for signature of OS and FlashApps, and therefore the seamless installation of arbitrarily modified OS (resigned with the deduced private keys) on TI-Z80 and TI-68k calculators, was made in 2009... but again, it did not get TI calculators banned from standardized tests for that very reason.
Nowadays, we're even fixing TI's bugs for them (there's an unofficial patch for the terribly unstable OS 2.53 MP for 84+).