Quantum computers are good at any problem where the size of the problem space scales exponentially with the size of the inputs.
More long-winded: when people talk about quantum computers, they always start with how each qubit can be in a superposition of 0 and 1 and then go into the Bloch sphere and woo! quantum! And it's all irrelevant. You can represent any number of isolated qubits states to arbitrary accuracy with a linear scaling to classical bits (e.g. to represent a single qubit state with double-precision floating point, you need 2 doubles = 128 bits). Since classical bits are approximately infinitely cheaper than qubits, you'd never bother with quantum.
What's important about quantum computing is entangling multiple qubits. The size of the state space of an arbitrarily-entangled set of qubits scales as 2^n. That means that representing ~50 arbitrarily-entangled qubits takes ~1 PB of classical memory.
The state-space scaling that entanglement gives you maps directly onto problem-space scaling. Factoring large numbers and many-body physics and chemistry problems both have in common that the possible solution space grows exponentially with the size of the inputs (how large a number you're trying to factor, or how many bodies in the n-body problem).
You're right that there aren't just a ton of problems that scale this way, but there are lots, and the dollar value of many of the applications are enormous (think designing new drugs with perfect fidelity).