The very influential critic Edmund Wilson, prominent beginning around 1920, is the apparent source of this prejudice - he despised Tolkien, C.S. Lewis, Lord Dunsany, Lovecraft - really all recent or contemporary fantastic literature being written in English (if it was old enough, Swift for example, it might get a pass).
I think critics' low opinion of some of those authors was deserved, because while those authors were masters of world-building, they were not masters of prose style. Their use of the English language feels flat and unimaginative. In spite of the rich detail of Middle Earth, Tolkien's prose, for example, is just as much uncreative aping of English epic writing as the Book of Mormon was Joseph Smith's aping of the King James Bible.
Valid points - Tolkien's prose is uneven in quality, and parts of LOTR are quite leaden, but the delightful Hobbit, for example, is entirely free of Saxon epic styling. But Wilson's dismissal was all encompassing, and drew few distinctions among these hated fantasists.
Lewis was better, over all, than Tolkien. But Wilson's attack on Dunsany, Lovecraft, and James Branch Cabell simply shows him as a pretentious fool. The fact that the greatest writer of the Twentieth Century Jorge Luis Borges (one of those Spanish fantasists, BTW) admired Lovecraft is enough to discredit Wilson.
One of the reasons I admire Wolfe's writing of the 1970s and recommend it to my friends who "don't read science fiction", is that Wolfe's prose in The Fifth Head of Cerberus and Peace is just as powerful as anything by Proust or Nabokov. Unfortunately, Wolfe's rich feel for the English language disappeared in the 1980s and some of his recent novels and short stories sound as bad any most genre fiction.
Agreed on all points (unfortunately, on the last). The The Fifth Head of Cerberus had a powerful effect on me, ditto Peace, and especially and forever The Book of the New Sun, which will in the fullness of time be remembered as a landmark work of the Twentieth Century. Soldier in the Mist is quite good, but with the odd There are Doors his genius seems to start to fading. I started the Book of the Long Sun but after the first volume, I did not want to be disappointed again.