You're both right. The bulk of the forces that attacked Washington (against orders I might add, the British ordered them to Baltimore but due to the lack of forces protecting Washington, Ross and Cockburn agreed it was too easy of a target to pass up) were the 3rd Brigade under General Ross and Rear Admiral Cockburn. In addition to the 3rd Brigade, there was a sizable number of British Militia, primarily from Halifax, but with some 300 from Upper Canada, mostly from Kingston and York. It was the ones from Upper Canada who bear mentioning since these were mostly the men who were at the Battle of York and were ordered to withdraw. It was their homes that the were vandalised, looted, destroyed and burned while Major-General Dearborn wrote in his journal it was despicable, but he was either unwilling or unable to stop his soldiers. The York militia, as one might imagine, wanted vengence. In addition to that, there was the matter of avenging the acts at Port Dover.
What I find the most interesting about the war of 1812 was not the later stages like the burning of the white house, which has already been mentioned, was white long before being torched, but the early battles (or lack thereof) at Fort Mackinac and Fort Detriot. Both of which were seized by General Isaac Brock and Tecumseh without a shot.