You realize that up until roughly two years ago (in the US at least) the majority of people who identified themselves as "libertarians" were left-wing hippies from Vermont who embraced social contracts but wanted the government out of their personal affairs, pot gardens and gun racks? It wasn't until the "Tea Party" under right-wing funding and Carl Rove style propaganda was the term "libertarian" bastardized to enable right-wing nut jobs to hide under the guise of a small government agenda. I am a libertarian and I vote Democrat; not because they are equal but because if you go by historical facts and numbers, the Democrats have consistently been the party of fiscal responsibility since the 1960's.
You're not crazy, the two groups are far more agreeable than you think.
Is it off topic? I blame Microsoft.