I believe this would be extortion not blackmail. Although, really, both are bad. You REALLY don't want to put your self in a position to be accused of either.
From a professional standpoint I would recommend staying away from any situation where you are making demands or threats to your boss. Such a situation will very rarely help you achieve your goals. More often it WILL damage your reputation, instead of being known as XYZ-good-thing, you will be remembered as the guy who threatened the company. Such extreme incidents have a habit of following you for years. It would be better to just leave than risk long-term damage to your reputation.
If you do want to change the situation, perhaps a less aggressive approach. Next time you are asked to install software product X, have the requester sign a statement that says they have acquired the proper license for this product (or product such a license). This removes the burden from yourself, it is reasonable. Lastly it has a subtle and non-hostile aspect of teaching others about licensing.
I would also recommend starting a license tracking system.
The basic idea here is that you don't want to assign blame. You DO want to move forward in a constructive and positive way, while still meeting your legal/moral standards. If your boss says that everything in use is ok, then don't press the point, but instead ensure that everything going forward can be documented as such.
As for the definition of blackmail, here are a couple of dictionary references (sorry no legal citations, but I am confidante that this would get you in trouble within the US Federal system.):
From dictionary.com
http://dictionary.reference.com/browse/blackmail?jss=0
blackmail /blækmel/
-noun
1. any payment extorted by intimidation, as by threats of injurious revelations or accusations.
2. the extortion of such payment: He confessed rather than suffer the dishonor of blackmail.
3. a tribute formerly exacted in the north of England and in Scotland by freebooting chiefs for protection from pillage.
-verb (used with object)
4. to extort money from (a person) by the use of threats.
5. to force or coerce into a particular action, statement, etc.: The strikers claimed they were blackmailed into signing the new contract.
Notice item number 5. To force or coerce into a particular action.
Another dictionary reference:
The crime involving a threat for purposes of compelling a person to do an act against his or her will, or for purposes of taking the person's money or property.
The term blackmail originally denoted a payment made by English persons residing along the border of Scotland to influential Scottish chieftains in exchange for protection from thieves and marauders.
In blackmail the threat might consist of physical injury to the threatened person or to someone loved by that person, or injury to a person's reputation. In some cases the victim is told that an illegal act he or she had previously committed will be exposed if the victim fails to comply with the demand.
Although blackmail is generally synonymous with extortion, some states distinguish the offenses by requiring that the former be in writing.
Blackmail is punishable by a fine, imprisonment, or both.
Notice paragraph 3. "In some cases the victim is told that an illegal act he or she had previously committed will be exposed if the victim fails to comply with the demand."
