ModSecurity (or any other WAF) can greatly decrease the number and kinds of attacks that actually make it through to your application. And like a good firewall it can alert you when you're under attack. If you do nothing else, put this in place.
You also want to make sure your app is solid, so head on over to DISA and see what the military recommends. They have Security Technical Implementation Guides (STIGs) for just about everything in your architecture: http://iase.disa.mil/stigs/app_security/index.html
Once you have things built, test! Use some of the open source penetration testing tools to see if there are any known vulnerabilities in your stack. Try it with and without your WAF in place.
Finally, if you really need to go the extra mile, it's time to shell out some cash for professional penetration testers. They'll have a tool belt full of open source and proprietary tools and the good ones will even do a static analysis of your code.