No it wasn't. A lot of money in Swiss banks is money held by the owners illegally, we're not talking about tax avoidance here, we're talking about out and out tax evasion.
Are you American by any chance?
The US certainly feels that way, but that's because the US has a retarded tax system that is based on citizenship as well as residency. That is, anyone unlucky enough to be born American (or get a green card or a bunch of other factors) is expected to pay tax to the IRS no matter where they go and live. Giving up your citizenship isn't so easy either, beyond the fact that you'd need some other country to take you in, there's an "exit tax" to pay too.
No other country except some tin-pot African dictatorship uses such a scheme. Unsurprisingly then, as seen from Washington literally every country in the world holds "lots of money from tax evaders", even though that position is stupid. Hence FATCA.
Now, the Swiss could certainly argue a different viewpoint to the one you just espoused. The Swiss have been wealthy for a long time, but the idea that banks are supposed to be some shadow police force is a very recent one. It dates to the US passage of the Banking Secrecy Act in the 70s and the Money Laundering Control Act in the 80s. Actually the whole concept of money laundering was created quite recently by the USA. As a social policy it's younger than most people are. Despite many extremely serious costs and side effects, these policies were then forced onto the rest of the world, through threat of financial sanctions in some cases.
The Swiss have always until recently had strict policy of strong financial privacy. So guess what - Swiss reluctance to sign up for the new fad of turning bankers into policemen suddenly means they're the bad guys. By the way, banking privacy in Switzerland has applied to their own tax collectors too. Somehow they still manage to collect tax, have a strong government, low crime rates, low inflation and low tax rates. Apparently their approach is not incompatible with civilised society after all!
This idea that the Swiss are rich exclusively because of some evil rule breaking is exactly the kind of absurd rhetoric that could lead to some invasion scenario, which is why the military uses it to practice with. But it's just not matched by reality. If you look at a GDP breakdown by sector you can see that manufacturing and specialised services make up a huge fraction of the Swiss economy. The biggest Swiss company isn't even a bank, it's Nestle. A big chunk of Swiss wealth comes from precision machinery, pharmaceuticals, IT, tourism and specialised financial services which are NOT banking (think industrial insurance etc).