I think what needs to happen is that people need to grow in their understanding that "literature" is just a genre of fiction, with no real claim to superiority over any other genre such as "romance", "western", or "science fiction". The Man Booker Prizes may claim to "honour fiction on a global basis", but really they only honour a particular genre of fiction, with a particular set of story structure, tropes, and other defining characteristics.
Top novels in any genre can often stretch or even break these types of "genre rules", and be praised for doing so. It is possible for a novel to be objectively categorized in more than one genre at a time but there are HUGE pre-conceived expectations for any novel based upon how that novel or author is "marketed" or perceived. Thus you have authors like Atwood working hard to maintain their status in the "literature" genre, and distancing themselves from the perceived ghetto of other genres.
Each genre (including "literature") is mostly made up of lesser quality works. 90% of everything is crap. But even if you look at the non-crap of each of these genres - aspects of each of these novels will continue to maintain some of the characteristics of their genre - and that can be challenging for a reader who is dismissive of the genre as a whole. Thus if a "literature" reader thinks (or suspects) a work is "science-fiction" they are probably more likely to think less of it when encountering features that are "science-fictionish" - and this goes both ways. A reader of a "historical romance" novel who generally dislikes "literature" may be put off by aspects of the novel that seem "literary".
So who cares? Well, I think society suffered when one genre is elevated in prestige or "value" over all others. If Chris is made to feel "lesser" because they enjoy science fiction, and Pat is looked down upon because they like romance novels, and Robin uses their love of "literature" to belittle others, good things do not follow. While there is little direct governmental support for fiction prizes, there is a pretty big media push for a number of "literature prizes" in comparison to other genres that tend to give such prizes more "value" in the eyes of the public.