I bought a Galaxy Nexus 2 years ago, and when I got it everything was blazing fast. By a few months ago, my phone was frustratingly slow. For a while I considered that I was just looking at newer phones and thinking "Wow, that's so much faster than mine" but just thought that it was the comparison that made mine feel slow, not that it really was slow. I finally decided my phone was objectively slow, not just by comparison.
A few weeks ago I 'downgraded' to Android 4.2.2 (Had been running Android 4.3), and turned off auto app update. (I had previously tried various ROMs with 4.3, but they were all still slow.) Now my phone is fast again. Maybe not as blazing fast as it was brand new, but I no longer feel like I'm ever waiting for the phone.
So, I don't think there is a real 'conspiracy' to slow down old phones, but I think that old phones *are* slower, they know they are doing it, and just don't care. Why would they? They think we should be happy they are pushing updates to us, but they don't think they should have to worry about the experience on devices they sold years ago. But, I have the ability to refuse these updates, so I can keep my personal phone usable for the foreseeable future.