However, when it comes to the very moment of starting, or the change of tempi, my start will always come too late.
Ah .. the trauma of remembering band practice:
Every conductor has a different style. The signal to start your part of a song that has already begun may be a small flick or pointing of the baton in your general direction, barely interrupting the overall tempo of the conducting, or if you have a dramatic conductor it can be a two-handed "picador going over the horns" gesture ... or no gesture at all.
Because the baton may be signalling to someone near the OP - in front or behind - but not the OP, the problem is discrimination as much as detection.
Also, it's not always a down beat. Changes of volume, extended notes and the final cut off of a long final note may be sweeping or tiny gestures sideways or straight towards the choir or orchestra.
Very few conductors will make big changes in tempo from what was practiced. No good will come of it.
In short, it might be more practical to start on the second note and drop out on the next to last note, paying attention to the parts of the production that immediately precede your bits so you are ready for it.