One way to at least try to make that harder is to have multiple implementations written totally differently (different code-bases, different languages, different development teams, different countries etc) so that you have different implementations both doing the same thing (harder to compromise multiple implementations like that)
Better yet, come up with a hardware box (open source, auditable and buildable by anyone but can be built secure and tamper-resistant too) that does the actual cryptography in a way that the userspace never sees the keys (again multiple implementations running on different FPGAs, MCUs etc, heck, build one that uses some obscure ancient CPU to reduce the chance the hardware is compromised)