That's pretty much the definition of a contract. Consideration (e.g. money) in exchange for service. If the contract has a clause saying the seller can refund the deposit and cancel the order, then the buyer agreed to that and too bad.
However, there is also civil law regarding business - the Uniform Commercial Code in the USA. That law preempts contract terms. The question then becomes, does the UCC forbid those kind of clauses. A little research leads me to believe it does not. That would make sense since companies like Gamestop cancel pre-orders all the time.
Given all that, Tesla are within their rights to say, "too bad, here's your money back".