The average manager at a large software company has no ability to blackmail you.

From my experience as a contractor - manager references are also highly overrated. You should be able to get references from other sources (such as former coworkers and the coworkers at your new job).

If this really bothers you and you don't need the money I suggest leaving the company earlier. Life is too short to get pushed around. Alternatively - if they bring it up again just say you probably won't give them a good reference either. They'll probably say something like "heh - I don't need a reference from you". Then you say, "ok, sounds like we're even".