The color inversion happens because e-ink screens are bistable -- the screen uses no electricity when it's displaying either the light or dark elements. In order to change what's shown on the screen, it needs to 'clear' what was displayed previously. This manifests itself as a brief 'flashing' of the screen. Incidentally, when it flashes is the only time that an e-ink screen consumes any energy. Conversely, your LCD monitor that doesn't require this flashing is constantly using electricity.
Also, the flash only takes about a quarter of a second. How long does it take you to turn a page in a paper book? I guarantee it's not too different. Certainly not enough to make an appreciable difference in your reading speed.