The victim may have losses that are difficult to document, but no less real. Proving those losses in court could be expensive for victim and perpetrator alike and time-consuming for the court. If a fine is required, it should go to the victim to compensate for any of those types of losses. I see no reason the fine should go to the government.
Fines should be large enough to discourage copyright infringement, but not so large that they encourage lawsuits. If that is not possible, then a fine is not appropriate and some other disincentive should be used.