It also has something to do with the competence of the freight railroads. Last March, I rode Amtrak's Crescent from NYC to Atlanta, a 17-hour trip conducted mostly over Norfolk Southern trackage once leaving the Northeast Corridor. We arrived in Atlanta on time, much to my surprise (we did have to wait for at least one freight traveling in the opposite direction in the dead of night). On the return trip the Crescent, already 12 hours into it's trip from New Orleans, not only arrived in Atlanta on time, but arrived and left Washington DC on time, and arrived in NYC ahead of schedule. The NS portion of this route is identified as a future high-speed line. I quite frankly can't see how they'd do it--watching the line from the last car on the return trip through Virginia, you would not believe how much the line twists, turns, rises and dips over a fairly large chunk of the route.
I've taken a number of trips NYC-Pittsburgh via Amtrak's Pennsylvanian (a 9-hour trip), also using NS tracks after leaving Amtrak-owned lines. Most of these were on-time. In one trip, the conductor specifically credited NS dispatchers with getting us out of Harrisburg, PA ahead of two waiting freight trains, and another year, we arrived on-time in Pittsburgh after tailgating a high-priority UPS freight for several miles. We were switched onto the adjacent track against traffic, and arrived in Pittsburgh literally racing alongside the freight we were behind only minutes before. This line (Harrisburg-Pittsburgh) is also identified as a future high-speed route. Frankly, the mountainous regions are something to behold when you remember that for the most part, the tracks go up and over them rather than through. Approaching Horseshoe Curve, you'll notice a road high up the mountainside on the opposite side of the valley. Then you notice that it's not a road, but it's the railroad you're traveling on. Then you remember that mile-long freight trains run on this line too. It's enough to give one pause.
The biggest delay I've had on these trips was about 45 minutes on one Pittsburgh-NYC trip, when our locomotive died pulling out of Philadelphia, PA on the way to NYC. No spare locomotives were available there, and we were transferred to the next arriving Corridor train from DC.
---PCJ