This is an example of the music I produce. It is produced using KXStudio and ardour with linuxsampler/hydrogen (running as an LV2 plugin via composite sampler)/linuxdsp/calf plugins. It shows that it is definitely possible to use opensource software to create songs. Of course i have also mixed in LinuxDSP (which is commercial) along with the calf plugins - they plays a big part in my sound.
Most of the money I have spent is on the equipment I use to record (my guitars/ tube condenser mic/X-Station/headphones etc). But I have also spent a lot of time and energy accumulating free samples from different sources and kitting them together(the drum kit is an example - it is a hydrogen based Drumkit using the Colombo Acoustic Drumkit with other samples (e.g. the snare) from different sources - all free). I also use the excellent composite sampler to directly plugin the hydrogen drumkits as a lv2 plugin into Ardour's midi tracks, so I don't use anything over ardour really. I use a cheap BCF2000 in Mackie emulation mode with Ardour.
I have also bought the linuxdsp plugins - I can honestly say that they are on par and sometimes better than commercial offerings (listen to the Linuxdsp Pultech EQ in action and compare that to the real thing - very close!) - the best part is that they are not restricted to linux - so you can use them where-ever. Also use the excellent calf plugins especially the saturator.
It works for the kind of music I do - (a mix of classical/classical rock/blues/jazz) and the fact that I compose/record/produce/sing my music myself, but I have felt the pain in the past and it has often taken me a lot of time to produce the things the way i wanted it to sound (You can see some of my older pieces as well on soundcloud - you can see that the sound does gets progressively better - it was part of a learning process of learning to use the tools and learning music production! I am currently working on a new track which uses the sonatina orchestra which is a free orchestral sample released under creative commons and i think that definitely sounds a lot better than my previous ones. Also Ardour allows for midi editing on screen - i.e.. i can see all my tracks side by side with midi at the same track resolution- its very useful when i need to line up notes across tracks. Other DAWs tend to have a separate window come up when you need to have midi editing (or they used to..not sure if that's the case anymore!)
If you go with a mac - chances are you will be doing what other people have already done and use the tools that they do - it does wind up costing more though- but if you are going to be producing music for other folks, time will be critical. Also there are probably more tools/options out there for the mac - e.g. I still can't find an auto tune equivalent for linux - however it is possible to run windows VSTs under emulation in wine as well - you can find videos in youtube.
The key thing with the mac is that if you run into problems..chances are someone would be able to help you solve them - I know a friend who absolutely swears by his macbook for music production and he says that the support is amazing. Linux based DAWs have also grown in that sense - the Ardour community is large but I just get the feeling that the mac might be a bit more mature - although this could be a case of the grass being greener on the other side. It will now be a bit of a learning curve, I am way too used to the way Ardour/jack and how my tools work now that I have invested the time and energy in getting to know them.
Also these tools have matured (i have been using them for over 5 years now). So a lot of the problems I faced in 2008 have been minimized. Suggest that if you do have the time, try giving them a spin with a simple project or something- spend a few weeks playing with it - if you like it use it. If you don't then you can always switch to commercial. The only thing you would have lost is time. That's what I did originally and didn't really look back after that.