There's a frequent misunderstanding when people talk about freedom with respect to the GPL. The concept of "freedom" is itself not well defined, and historically there are at least two competing and somewhat opposing definitions, "positive" freedom (which is about maximizing the amount of things people are able to do) and "negative" freedom (not interfering with things that others want to do). The GPL is primarily concerned about the former, and your complaints are about the latter.
The goal of the GPL is that everybody can use the software for any purpose, and learn how any changes and modifications work; this is seen as a requirement to increase the amount of things that can be done with the software, guaranteeing that it can be adapted to any hardware or platform, with no commercial secrets getting in the way.
In order to achieve this goal, the GPL doesn't come "free" (as in "gratis", i.e. no cost): it has a cost that you must pay if you want to use it; but for anyone willing to pay it, there are no further restrictions imposed by anyone, for any modified version. In your case you *could* have merged your application code with the GPL library* for any purpose**, but you should be willing to pay the cost, which is to release your own source code when you republish the software. So, your negative freedom is reduced (you are forbidden from keeping your version of the software hidden and publish just the binaries), but the positive freedom of the system is increased - overall there are more people who know how to use your modifications and adapt them for other uses, which couldn't happen if you kept your modifications secret.
The expectation is that by adding contributions from many users to the pool of knowledge, the whole society sees an increase in the amount of possibilities to use the software ("positive freedom"); it's the same principle that motivated the patent system in the Renaissance. The "release what you know" cost is intended to publish knowledge that otherwise would not be shared, and thus cumulatively improve the whole system. Now, there are valid concerns that the upfront cost may instead work as a disincentive to participate in the system (both with patents and copyleft software), but that argument doesn't make less free.
* Assuming the original license and copyright law allowed it. (This is why the FSF recommends using only GPL-compatible licenses).
**(Including selling it, although with FLOSS software this typically only works once for each release).