Comment Re:What is this bullshit? (Score 5, Interesting) 92
Odds are they looped the network (I have seen it happen on other wan router upgrades/wan wiring upgrades) it cause severe network havoc and the connections pass thousands fast.. I have also see the network issue cause disconnects and reconnects on the backends and each of those new connections create new processes and use ram and it only takes that going on for a while to crush the backend's cpu and ram (because the old processes do not go away fast enough).
If they managed to loop a network that uses a routing protocol then I'd be very impressed. They are using L3 links, right?
I've dealt with L2 networks that had loops leading to broadcast storms, but the last time I saw a network that used multi-hop L2 across WAN links was a legacy topology that derived from the prior Novell Netware network, with one TCP/IP router across the whole organization for the enterprise zone. It frankly was a shiatshow because even though each site had its own VLAN, the L3 router could end up swamped if there was a loop at any of the nearly 100 sites. For reasons I still don't understand the prior network engineer had disabled spanning-tree to the access port level at the sites as well.
In a L3 network if the routes are configured statically then I could see it being possible to introduce some kind of routing loop, but we've had reliable routing protocols for decades now. We can even construct the network with an overlay/underlay model using older technologies like MPLS or newer ones like EVPN if one wants to get fancy, allowing for the underlay to handle the convergence and other control-plane messaging with the data-plane traffic nicely segmented on the overlay. Juniper supports this, and if there's a mix of HPE and Juniper gear in this network, the HPE stuff should still support it.
So that this happened on a Friday afternoon is unconscionable. This is stuff that should be reserved for a minimal-traffic period if an actual downtime window isn't possible, when as many of the redundant systems (ie older physical radios for controllers to use) are up and on standby. That this happened during one of the busiest times of the week is ridiculous, and the timing alone calls into doubt the claim of what happened.