System and numerical libraries and compilers are of course written specifically for the machine. But user-level apps (and a lot of scientific computing uses finished apps) are ported across multiple systems.
Portability is not as big an issue as it was a generation ago, as most supercomputers basically are Linux machines today, and made to more or less look like a typical Linux installation from a user-application level, with a POSIX API; pthreads, OpenMP and OpenMPI; a standard set of numerical libraries; and often even gcc-compatibility in order to minimize the effort of porting. A notable exception is GPU-based machines (that are in the minority today, despite the OP assertion); they don't have a common API to write for, so using them is substantially harder at a user-level.
And at a user level (but unike system libs) porting or coding time very much matters. Let's say your project is going to need a month of wall-clock computing time during the course of a year or two. If switching to a GPU-based system would shrink that by 50% - two weeks - then the effort to move your model code, app, and libraries had better take less than two weeks of work or you're going to waste project time, not save it.