I also use 2 displays and 2 KVMs and a dual switch monitor, All my computers accept my primary one are connected to a KVM, This is then hooked up to a VGA doubler, one output is then sent to my second display (on the right), another output goes to my second KVM, which has the primary display of my primary computer on it. The secondary display of my primary computer is also hooked up to my secondary display via the input switch.
This setup has the effect that any computer display on my primary monitor as keyboard and mouse on it, this saves alot of confusion. My secondary display can either display any of my other computers or the secondary one of my primary computer, this allows me monitor any computer I wish with my secondary display, if my primary is the one I am wanting to monitor I can still to that via the secondary display, while using another on the primary display. This setup is worth while as I do have 7 computers in my study though most of them are not turned on at the same time (most of them are a san/fibre cluster testbed).
The KVM I have most of the computers plugged into, is infact an cat5/ipkvm, this means other computers arround the house can have their displays routed back to it, also have satelight adapters, I only use one cat5 to each room to do this. Oftain when I want to use my laptop at home I plug it in down stairs, out of the way and then work in my study.
I also have a TV to the left of me, which I am going to get rid of and replace with another monitor, which can take an input from my cable box as well as a secondary display from a media computer, this monitor will do PiP as well.
Of course adding to all of this I have vmware running on my main server so I always have access to a working windows image (and that can do remote displays as well), as my primary computer like many other peoples is aways in some stage of being broken!