Sure, appealing to authority is unscientific but to assume there is no qualitative difference in the opinions of the two groups simply implies you think that all opinions are equal
It's not that all opinions are equal, but that blind faith is blind faith. Science is great because you can do the diligence and confirm the opinion, or at least understand the argument. But until you do that, the difference is as yet immaterial.
It amazes me how many people have strong opinions about issues they don't understand. I take a lot of things on faith: pretty much everything in my life that's both unimportant and uninteresting. But I don't have strong opinions on those things - I know I can't back up my beliefs. But for something I have a strong opinion on, say relativity or evolution, I can explain the science, qualitatively and with simple math. I understand the predictions made, and how they are confirmed, And, most importantly, I understand the arguments of the skeptics - I don't dismiss them out of arrogance, I understand where they're coming from and why they're wrong.
Similarly on issues like abortion, or normative ethics, I can explain in detail why anyone who's too certain about their stance just hasn't thought deeply enough about the issue, because those are areas where there's just no way to reach certainty except ignorance.