Comment Re:Just get used to it (Score 4, Informative) 53
We had an article here about power grids pulsing due they suspect to large numbers UPS switching in and other systems leaving sleep states on the hour running cron jobs and the like all synced very to very precise time via the network.
Most software testing has focused on 'correctness' and but 'emergent behavior' and specifically how small seemingly innocuous changes to software will manifest not request volume / batch size scale but 1000s separate servers scale. Microsoft leaned about this last week with patches that got it wrong on the single system scale worked well for handfuls of CPU cores not so well on systems with 10s or 100s of concurrent thread support, and at "hyper-scale" where fail-over and back-off behavior lead things to cascade.
However how do you test that, even for Microsoft the economics having a few extra data centers fully kitted out with their latest hardware platforms for 'testing' is tough to swallow and even if they had that how do they simulate the traffic? You can do experiments like 'well if we really light up this load balancer, we should see it shed requests to the others...' but that does not tell you how all those client workloads are going to react.
Nor can you lab this out easily either because you don't have as much real concurrency, even if you had VM representing every physical machine you own.. you're going to be serializing things in a way that don't happen in the real world where there are truly independent systems running.
For all the testing you can do, at that scale humans or maybe AI can really only pontificate about what might happen what the odd failure modes could be..We just can't know. The thing to remember though is even if you build you own separate systems you are going to have failures and downtime, for most of us more failures and more downtime than "the cloud" but you get a known worse case, that being completely rebuilding should be no worse than the initial build. There is probably some serious soul search to be done about just how tightly coupled and where information systems really belong. Now none of that really happen of course because that might cost real money..