Your understanding that you have to be a suspect, or that you have to be on trial, is incorrect.
There was a case that went to the supreme court where a witness - not the person on trial, but a witness - took the fifth. The prosecution gave the witness immunity. The witness then testified, and none of the testimony was incriminating.
The defense brought this up on appeal trying to claim that because what the witness testified to was not incriminating, the witness should not have been able to take the fifth.
The supreme court ruled that the witness taking the fifth was fine. The standard was not that the person taking the fifth was the one on trial, nor that it was actually even a trial, nor that the person testifying was guilty or innocent.
I don't remember off the top of my head what the exact wording was, but it boiled down to whether or not the witness could reasonably think that truthful answers might be incriminating.
In this particular case, it was a murder trial, and the defense was that this witness was the one-armed man. It was entirely reasonable for the witness to think that the defense might attempt to ask questions for which truthful answers might be incriminating, despite the fact that the witness asserted innocence and when given immunity, did not actually give incriminating answers.