"I'll need to be able to pull by date or by a number of key fields"
So, in other words, you have already decided on key fields. If you use a database, this has things call index's, that can search billions of rows for a key field in a fraction of a second.
If you don't use something with INDEX's then you can't do this.
Where has this idea that Databases can't scale come from? - The world runs on Database for heaven sake. Do you think when you take money out of an ATM, its going to MONGODB? - And yet there are millions of ATM's and you can take money out of your VISA account in almost all of them anywhere in the world. That is called scale.
How cool is that!!! If that was me, I would make sure they didn't put the skin back on, and added a few blue flashing LED's for additional affect.
"It will not be used as a disintegrating death ray, but to study symmetries and invariants. This is much more interesting..."
Oh, no its not!
I wonder, how much Co2 has been released into the atmosphere, with this bug present on millions of computers, over decades, causing PC's to eat more electricity than they should.
Its also more formally called TDD. Create some code that tests the suitability of the existing code to solve the problem. Then randomly change the code until it passes the test, and all the others. Repeat, rinse, etc.
Its a commodity. Like gold, or silver.
Apple don't keep changing stuff around in their Mac OS, unlike Microsoft do with windows, so my mum can keep up to date without getting lost.
Yes, I do have to fix stuff sometimes, but that doesn't include removing dozens of spyware etc.