Javascript doesn't attack a browser in the classical sense. The way you cause damage with JS is poisoning the browser's cache. So you add something sketchy to the cached version of a given webpage.
The classical route of this attack is a proxy that injects code to cache sketchy objects on top of the cache of any page visited. The cache expiration is set to something ridiculously high, so it's not removed without clearing the cache.
So for example injecting an ad that wasn't there before into youtube, slashdot, etc. Every time the user loads the page they load your ad, and get you an impression.
So yes this is strictly limited to browsers, and even within the browser is quite limited.
I asked if you meant Java, because there have been attacks in Java that can escape the browser sandbox and modify system files. Potentially java could be used to infect a server via a means besides a browser.
Flash is not on servers, no one checks email on servers, and no one views word/excel documents on a server. Word/excel files may be "viewed" on a server, but that would be for processing. In which case they would be accessed using something like the mono interop API (C#) or Apache tika/POI. So embedded bytecode wouldn't be executed.
I suppose these things could happen on a windows servers, but if you're admin is browsing and checking his email on a server...ffs