I often develop for VPSes running CentOS and DirectAdmin.
So i got a VPS like that for ~€14 /month. I upgrade pretty much the LAMP stack using DireactAdmin custombuild, and have VPS snapshots to easily roll back to in case i broke something. I need to maintain the VPS though, but my hoster informs me about (and how to fix) the latest critical bugs. Its simple and enlightening.
Developing this way is quite versatile too; i make a user per solution. I can backup that user and import it on other servers (production) or just archive this username.tar.gz file. This backup bundles relevant files, databases, accounts such as email or ftp, even DNS records if i want to.