As far as I can tell, this is a non-cryptographic use of hashing.
Git uses sha1 hashes to identify everything.
A (possiblly signed) tag references a commit by hash
A commit references a tree by hash
A tree references a list of files and subtrees by hash
If a commit you fetch references hashes you already have the files for in your local git tree they will not be re-fetched, the existing ones will simply be used.
The whole point of git is to be distributed, so it should be safe to fetch commits from untrusted sources, inspect them and throw them away without worrying that they will change the meaning of commits you later fetch from trusted sources. It should be safe to download commits over an insecure connection and then verify the commit hash (either by a signed tag or by checking out of band) to ensure that the commit hasn't been tampered with.
The latter part of linus's mail is quite a well-reasoned argument as to why the current attack on SHA1 isn't too big a deal for source code repositories.
If a "distinct chosen prefix" collision attack shows up then the risk gets much higher. For MD5 it took about 2 years to go from a basic collision attack to a distinct chosen prefix one.