"Let he who is without sin cast the first stone."
This is a Sophists' phrase, and it is an unnecessary constraint. Why would a man who failed to pray on $holy_day be prevented from punishing a rapist? Both men are sinners.
I, personally, do not worry if the judge is a sinner. The only requirement is that he judges fairly and by the law.
It's not meant to be a 'phrase' in this context, it's part of a parable, an argument structured in a story, and you have to understand the story. The law on the books at the time was that adultery was punished by stoning, and so "fair and by the law" judgment being put into effect would mean the adulterous woman would be pommeled with rocks until she died.
Jesus was not concerned that the wrong sort of person was going to be doing the stoning. His concern was that the woman was being stoned at all. So he reminded her accusers that they were in equal standing with the woman as far as being guilty (in God's sight, if not that of the local officials), and coincidentally made it very difficult for them to carry out their task. (If they threw a stone and the others took it as a serious claim of sinlessness, that person could be in for some unpleasant repercussions themselves.)
What you are meant to take away from it is insight into Christ's character -- forgiveness, providing redemption, showing compassion -- and the associated moral philosophy. The idea that everyone does wrong, that no one can boast about being better than others, that everyone needs God's forgiveness, and as recipients of that forgiveness it is hypocritical if we don't show that same mercy to others.
When someone says, "Let he who is without sin cast the first stone," they are typically inviting a bit of similar reflection.