As a founder of two startups we're been here multiple times. Here's what I've found.
Google (email and docs) works okay for very early stage (engineering only - no sales/marketing people - little need to communicate outside of the company).
As we got closer to launch and hired more outbound people we moved to using Hosted Exchange (Intermedia.net). Outlook is the driving force here, I have code to write and don't want to spend my expensive time fixing email/calendar/desktop support issues.
For Office applications we joined the Microsoft ISV program where we get 10 licenses for all their office products for about $400 per year. That also includes MSDN access so engineering can use Visual Studio.
Engineering does not use Office, all internal engineering documents are on the hosted Wiki (Atlassian) - but the hosted Exchange comes with an Outlook license so developers use that. I will neither help or hinder the use of anything else.
Everyone uses Windows on their laptop - using VMware Workstation to run the Linux VMs used for development.
We run the entire business on hosted services (Intermedia, Atlassian, JungleDisk (backup) and VirtualPBX). Our monthly bill is ~$600 for a 25 person startup - core engineering is now about half the company.
We have ~60 servers - but all are for dev and test, there is no "IT overhead"
The issue is not that you can't make it something else work - but why ? Unless you're developing an office or email software its just not a good use of your expensive (unique) resources. The goal of your company should be to efficiently sell more of your products to people that are likely using Microsoft products (at least the decision makers). So for maximum interoperability and profession appearance use the products your customers are using.
(I use a Mac, but I cannot use it for anything for external communication (PowerPoint, Word etc) - somethings just look different to the Windows version (fonts, text positioning etc). Not all the time, but enough to make it unusable from a professional appearance point of view.