Any heavily accessed server would have to keep track of source IP. If the knock is abandoned by accessing the wrong port nobody would be allowed to enter as a new knock would start before the previous was finished.
And if you track the source IP it is trivial to only allow connections from that IP to the server.