Yes, you can vote for a smaller party. But to do that without actually supporting the policies of that party is lunacy. I imagine you are in agreement with the Libertarian party's platform (though it's hard to imagine anyone literally wanting their 'essentially no government' platform enacted. But presumably the swing, potential 3rd party vote this year is left-of-center Sanders supporters who basically (grudgingly) accept that the Clinton/Democratic platform is a pretty good approximation of Sanders' 'pull the Dems to the left' goal at the start of his campaign.
Some of these voters, though, either think Clinton won't enact any of that platform and/or that voting for a candidate that they dislike and/or distrust is worse than having a candidate they seriously dislike and distrust actually win. Okay kids, it's not about how you feel - it's about what you want to happen, and the policies you presumably care about. And as far as Clinton backtracking on the platform, that platform contains plenty that Clinton didn't have to be pulled into embracing - and probably some that she's glad to have been 'forced' to embrace.
Now maybe some think they're pursuing a fantasy strategy in which Trump wins and is so awful that a true leftist wins in 2020. Fat chance - since Dems tend to trend rightward after a big defeat. And of course, the next 4 years - and a couple of lifetime Supreme Court appointments - will make a mess that's even harder to clean up.
It's the whole 'Nader as spoiler' thing all over. Yes Nader and his apologists will claim robotically that "If Al Gore couldn't win his home state, it's not my fault", or "If it hadn't been for the ballot problems in Florida, Gore would've won anyway". But they conveniently discount the effect of Nader's running around the country saying "your choice is Coke or Pepsi" to an audience that missed all the nuance and assumed that meant electing Bush and Gore were essentially the same thing. Nader doesn't and didn't believe a Bush win was the same as a Gore win, but his message implied it. Nobody knows how much that detracted from Gore, but I suspect it had a greater effect than the direct Nader vote.
Yes, the two party system is a problem. No, voting for 3rd party spoilers within the 2 party system will not fix it.