The majority of the arguments against building codes are for people building on their own property for their own use at their own expense.
Yes, building codes are far more important during purchase or rental. In that case, if building codes didn't exist, then full disclosure and understandable ratings of building materials and practices would at least be necessary. If a rental agreement was required to say "Electrical safety rating: 0 Stars, Completely Hazardous via ASDF rating 2013" it wouldn't prevent anybody from doing anything, but would strongly disincentivize people from renting there.
I'm not promoting either direction, but that's at least how a complete argument from the libertarian side would go. Many of them are big on contract law, trying to ensure that contractual exchanges and agreements are properly informed and consensual. Of course, there are the "Well, you should have known better" side of things, but when it gets down to details like many take the route of making sure the legal agreement is more sensible