I can think of plenty of consistent reasons for this (not saying I necessarily agree or disagree with them)...
First, this should limit the risk of coercion. The main reason that people (the internet) seem to cite is that the self-incrimination clause was put in in response to an English history of torture (or other coercion) to elicit guilty pleas. Writing a law to restrict unwanted behavior is pretty straightforwardly reasonable. Also, it should be easy to accept that the risk of torturous coercion of a person suspected of being guilty of a crime is less than the risk to someone (or many someones) suspected of witnessing that same crime, if for no other reason than reviewing such instances in history.
Next, the risk of false testimony and the usefulness of self-incrimination due to personal incentives. The incentive for a guilty party on trial to lie is very high, but the incentive for the third party is very different. A witness, typically offered protection from prosecution for self-incrimination in these cases, has no supposed social or personal reason to lie (although obviously they could be biased for some reasons). They are, by not testifying, at worst facilitating a miscarriage of justice, and at best simply choosing not to assist in a process that is all but undeniably in the public interest. Note that this applies if the testimony leans towards innocence or guilt of the accused.
This social responsibility aspect is another justification for punishing witnesses who refuse to testify. It is in society's interests to find and punish criminals. Witnesses are a necessary component of that process. The more unbiased and uncoerced witnesses and evidence that can be supplied, the more confident we are that justice is being served accurately. It is therefore reasonable to establish an incentive to promote testimony. The threat of jail time is the incentive we currently have. Again, the value of witness testimony almost certainly is more valuable than testimony of the defendant, given the incentive to lie, so adding a disincentive to remain quiet can easily be seen as balancing the value and likelihood of receiving good testimony.
Beyond that: you're oversimplifying. Your example is merely a yes/no question of guilt, but, at least in the US, a defendant is in fact required to enter a plea. Guilty, Not Guilty, no contest, and variations (such as not guilty by justification) exist. The defendant must take some stand on that point, but other details they can keep to themselves. This isn't really a justification of an answer to your core query, but it is an important distinction, and related to...
Oversimplifying part two: your example pretends that if a defendant pleads the fifth nothing happens while a witness not testifying goes to jail. This is apples-to-oranges. Witnesses CAN take the fifth amendment, for one thing, which is different than not testifying when they are not self-incriminating. The sixth amendment is actually the one that causes witnesses to be held responsible for not testifying, not the fifth. That the two rules are based on different portions of the constitution is not justification itself, but it does lend some clues to understanding why we've balanced things this way. Remember that all a witness needs to do to avoid punishment is to answer questions, presumably with the truth. If the witness believes what they say will be self-incrimination they can plead the 5th, but this is a very different scenario than the one you're depicting.