... since the software needs to be verifiable ...
The software does NOT have to be "verifiable". It just has to be thoroughly tested, and in practice, shown to be better than humans. It doesn't have to be perfect, it just has to be an improvement.
Only trivial programs can be mathematically verified. Even for mission critical programs that make life and death decisions, very few can be proven correct. And even then, are you sure you trust the proof?
There are techniques for making ANNs more reliable. One technique is "boosting": Independently train two neural networks to solve the same problem, then train a third network on the sample cases where the first two disagree. Then use 2-out-of-3 voting. This can reduce the error rate by a factor of 10.