Actually, you are slightly wrong. The three options you suggest apply before you are in a situation of copyright violation: when you are considering distributing your project, you have to do one of these things.
But after the copyright violation, it is too late. If you steal something, get caught and give it back, you still go to prison. The same applies to any kind of wrongdoing: undoing it after getting caught does not avoid the punishment.
The GPL has an explicit provision for violations: (emphasis mine) “Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License.”
In other words, if someone violates the terms of the GPL and gets caught, they lose all right to the software, and have to beg the copyright holders to get them back. The copyright holders may be satisfied with simple compliance, but they may require extra good will proofs.
Of course, if the infringer does not want to comply, the only option becomes to sue them. And the judge would not order compliance, only damages.