It's great work, very fascinating. The speed can probably improved up to a point but it could be a valuable technology based only on the energy per bit switching event, since this sets the energy limit for a given piece of computation.
A more important challenge is integrating this into a circuit and figuring out how to get electrical signals into optical switches based on the output of an optical switch. Transistors are great since they are electrical input and output, and with complementary logic they only draw a leakage current except while they are switching. But a voltage controlled optical/plasmonic switch is strictly speaking a transducer and so you need it paired with the inverse transducer element to create cascadable elements. And you need something supplying the light. And while there is a small switching element there is a (comparatively) huge waveguide feeding structure. So solving these issues and making sure that you don't get huge energy losses when you add the pieces surrounding the switch element, and figuring out how to combine large numbers of elements onto a small chip is the bigger challenge for bringing this into a computing application. Or maybe it can make a very energy efficient optical modulator, not used for computation.