Because they don't just operate n the visible spectrum. Using infrared, they know people are in specific locations in a house. In England, where this practice is common, it is a regular event for houses to be raided for drug "grow rooms" just because of an unusual heat signature. Often, it's just been a poor insulation job in winter, and you DON'T get any apology.
Worse, in the UK for example, if your electricity usage unexpectedly increases, you'll have the same raid and lack of apology.
It's the combining of this archived video data with other data sources that makes this an intrusion that goes INSIDE your home and crosses the line to being an unlawful search, if you hold the view that infrared is not "in plain sight" as many do.