An ip kvm will do exactly what you want. They are used in data centers to control machines throughout the building. You can get a used Raritan on ebay for about $200-$250.
As others have pointed out, hdmi video is multiple gigabits, so you're not going to have Bluray quality video, or gaming FPS, without a dedicated tables. If you want that kind of video, the standard is called hdbase-t. You can run it over cat5e or cat6.
You do NOT need to drill new holes and go through the same trouble you did when you installed the first cable. Simply tie a string to the existing cable, then pull it out, leaving the string in it's place. Tie on another cable and another string and pull it back the other way. Now you have two cables where you used to have one, AND you have a string ready to go for adding another cable. Next time, pull a string along with any cable you pull. If the holes are too small for two cables, just enlarge the existing holes a bit. Don't try to force too many cables into too small of a hole. Just take three minutes to drill it out a bit larger.
Ps - for any new holes s, don't drill through brick, drill through the mortar.