You tried to annex Canada in 1814, so we beat the crap out of you and burned down the Whitehouse.
Unfortunately due to commitments defeating Napoleon at the time, we couldn't commit enough resources to take back our American colonies so we settled on leaving the borders where they were before you invaded British territory.
The British tried to enforce trade restriction on the United States with trade with a verity of European countries and were boarding American merchant ships and pressing their sailors into service in the British navy against their will, at the same time they were funding and supplying native tribes to fight again American expansion westward. So the United States declared war on the British Empire.
Additionally the United States never seriously tried to take Canadian territory they sent a token force of untrained militia Under the command of William Hull to keep the British troops in Canada out of the way.
And it wasn't the Canadians the burned the Whitehouse it was British troops that came down out of Canada.