The fundamental problem for the A380 is that twin jets are cheaper to operate than quad jets. The 777 may not carry as many passengers, but it has lower cost per passenger-kilometre than the A380, or indeed the 747 (which isn't so different in size than the 777.)
I have seen this stated many times. However, I don't really understand why. For example, this article states "those newer, more reliable [twin jet] engines have also been bigger and more efficient" but doesn't say why jet engine companies aren't also making more reliable and efficient engines for quad jets, if it is all down to newness.
I've looked online for an answer, but generally I just find speculation from people who seem no more knowledgeable than myself.
I can give an argument why quad jets should be cheaper. A plane needs to have sufficient thrust to take off and climb out after a single engine failure at the worst time (just after it is too late to abort take off.) If we call this thrust T, then a twin jet needs each engine to be capable of producing T, so it provides 2T thrust. For a quad jet, after a failure it needs T, so each of the remaining engines needs 1/3T, hence when fully operational it need only provide 4/3 T thrust. So a twin jet must be much more over provisioned than a quad.
Speculations I've seen include that four engines are extra complexity (but those big twin engines need a lot of extra complexity to be so big), extra maintenance costs, that engines disrupt lift, so you need bigger wings (and more weight) to make up the losses of having two extra engines, that with engines further outboard you need more structural strength in the wings (hence more cost and weight.)
I'm not saying these reasons are wrong, just that nobody I've seen making those arguments has convinced me they know what they're talking about. In replies, if you have expertise, or are citing someone with expertise, please make this clear.