Full disclosure - I am a founder of a startup that develops an open source automated trading platform targeted at institutional investors.
As was mentioned in above postings, there are a series of open source tools available to bootstrap your trading system development:
And then of course there's my company Marketcetera - we build on top of a lot of the tools mentioned above and others (ActiveMQ, MySQL, Ruby on Rails, QFJ, etc) to provide the basic underlying platform that institutional traders (think quantitative hedge funds) can use to build their proprietary algorithms and start trading. After implementing a few trading systems in a row ourselves for various trading firms we realized that there was an obvious need for an open source trading platform so that people wouldn't have to reinvent the wheel and write systems from scratch every single time.
To answer the OP's question about which commercial firms use FOSS: - a lot of proprietary trading software is implemented on top of OSS - JPMorgan famously built their trading GUI [PDF] on top of Eclipse, and Progress Apama is built on top of Eclipse RCP as well.
Not surprisingly, most trading applications are very Windows-heavy (although
quite a few companies have Linux clusters, and some exchanges run on Linux as
well). Most of the apps that your broker will provide for you to trade with are
Windows-only (such as Bloomberg, Goldman Redi, MicroHedge, etc), and a lot of
the APIs available from vendors are
We know that flexibility is at the heart of any powerful trading application, and we think the open-source model maximizes the ability of our users to control the application. Some think the open-source model is antithetical to the secretive finance industry, but we see it as the perfect fit.
My sister opened a computer store in Hawaii. She sells C shells down by the seashore.