If you're going for Boost, you're diving into a sea of angry template-metaprogramming. What's seriously needed in the world of mathy software is more really good computational geometry packages. What doesn't exist right now (to my knowledge) is an open source implementation of M. Held's algorithm for generating generalized voronoi diagrams from poly-line curves (arcs and lines). Open source manufacturing is picking up and the algorithms to actually take the level sets, generate toolpaths, and all this are just starting to bubble up. This would be an excellent domain for an applied mathematician to dive into. Just make sure to try and get someone to watch over your shoulder to make sure the API is nice.

I smell an SGI naming convention, i.e. RealityEngine

I really hate to say it, but basic on micro controllers has been done way way way too much. I think it would be far more appealing if you could write a small JVM for it (pretty easy), or have the user write their programs in C where the firmware serves as kind of a network bootloader, and you release the tcp stack and web server stuff as a library.