... and I can safely write that there is no way you will ever achieve anything comparable to gmail.
You can try:
- squirrelmail, ugly and so last century
- openwebmail, old-fashioned Perl webmail, not maintained any longer
- zimbra mail, lots of functionalities and fancy features
- roundcube, decent but nowhere near what you're hoping for
Spam control on the server side is going to be an issue. You will have to use a combination of solutions (e.g. custom sendmail configs, RBL/XBL blacklists, spamassassin, greylisting, procmail rules, smf-spf, j-chkmail) and it will take quite some time and effort to get everything fine-tuned.
For anti-virus, clamav works well
For IMAP I found that dovecot does a decent job. If you want to fetch from remote servers into your own server then fetchmail can do the job.
Usual security considerations apply - patch early, patch often.
You will spend long hours maintaining this, highly recommend using a log colorizer to help watching logs e.g. ccze
In the end you will feel you got a half-baked solution that doesn't even come close to comparing to gmail in terms of functionality, ease of user interface, security and spam control. But hey, it'll be your own stuff.