Take for example a GPLed game. Now somebody comes along, takes all the graphics and releases a proprietary game with them. Is that a violation or not?
Absolutely a violation of all GPL versions.
What if he doesn't actually include the graphics, but loads them from the web or straight out of your git repository [...]?
Possibly a violation of GPLv2. Probably a violation of GPLv3. Causing the game to download graphics for the user's benefit is not distribution on your part of those graphics. The original developer is distributing the graphics by way of Github, etc. The original developer may also cease that distribution at any time (or rename a directory) and break the game.
The question (for a jury) would be if your game constitutes a derivative work of the original such that distributing only your portion without the images isn't actually distributing a distinct creative work in its own right but only distributing a portion of the derivative work created by combining the parts. If your game could load graphics from additional sources than the original and still do something, that would significantly decrease the likelihood that it would constitute a derivative work. If it was completely useless without the single other game's images, then it's probably derivative. Additionally, requiring the user to take affirmative action to download the graphics themselves and comply with any license would greatly help your standing. Requiring them to run `git clone ...` locally and informing them that what they're cloning is covered under GPL should do the trick, same as many distros require you to download tarballs of Java separately from their package manager.
Larger possibly that this is a violation of GPLv3. GPLv3 closes some of the service provider based loopholes on what constitutes "distribution." I don't know if this would full under that language. This one might be "up to a judge," but a lawyer may tell you differently with respect to how clearly this does or doesn't fall under GPLv3 language.
[...] or their fork of your git repository?
Absolutely a violation of all versions. Now you're the one distributing it and must comply with the GPL in order to have a right to do so.
What if it's not graphics, but interpreted code code?
Copyright makes no distinction on the medium. Both graphics and source code are well established as creative work covered by copyright. You may NOT copy such works without a license. GPL is a distribution license not a use license. Distribution of interpreted code is still done by the original developer in this case. You could NOT distribute modifications of the code yourself. You *could* distribute patch files to be applied on the user's computer at runtime. You may run into a situation where extensive patches must contain so much of the original files as diff context that the diff is arguably a derived work. "Up to a jury" in terms of where that line is for the patch case specifically.
At what point do the assets require the rest of the code released under GPL?
When the final work becomes a deriviative work under copyright law it falls under GPL. GPL additionally specifies that you may not link a work against GPL code and distribute it if the combined work restricts distribution beyond what the GPL provides.
if you use one of their GPLed libraries, your main app shall be released as GPL as well.
False. If you link against a GPL library, you must distribute your app under a GPL compatible license. If you're trying to distribute closed-source binary, you can't do that. If you don't like GPL, Apache, BSD, etc. are all options for your own code's license.
Shall APIs have copyright or not?
That's the key question, and it's what makes it clear FSF doesn't want it both ways. Until Oracle tried suing Google, it was long accepted that API's aren't copyrightable. Even Microsoft was content with letting Samba reimplement all the Windows file sharing API's. FSF is quite clear that API's are not covered by copyright and can be re-implemented by anyone who wants under any license. It's those implementations that represent a creative human endeavor that is protected by copyright.
The only group that wants API's to by copyrightable at this point is Oracle because Google used the fact that they're not to avoid paying substantial fees to license J2ME and instead implemented the same API's in their own code. Oracle's Java licensing attempted to place a restriction on what kind of hardware you could run Java on without paying. PC's = free, mobile = pay $$$$ for the same code. That's a lot more trying to have it both ways than anything FSF has ever done.