Passphrases are not harder to brute force. In general if you have 26 random characters its hard to brute force.
Passphrases encourage the use of numbers, capitalization, longer passwords, and punctuation. If the common password is all lowercase letters and maybe digits, your looking at a search space of (26+10)^k for a password of length k. If you throw in the 30 or so punctuation marks, and capitalization, the search space is (26+26+30)^k for the same length of password.
Given that so many people use lowercase+digits passwords, I'd be inclined to think that anyone brute-forcing a bunch of passwords would stick to the (26+10)^k search space, and therefore leave yours uncrackable. If they're just going after yours though, all bets are off, but then you should probably be using some uber-fancy authentication scheme anyway.