The three V bombers were designed for high altitude operations. They were switched to low altitudes due to the threat of SAMs. The air down there is denser and wears aircraft out faster.
The Vickers Valiant wore out first, with it's long flexible wings.
The Avro Vulcan has delta wings which are less flexible and allowed it to operate at low level for far longer.
The Handley Page Victor lasted longest because it was converted to reconnaisance and tanking and stayed in thinner air. It's enormous bomb bay made it suitable for this job.
The B52 was never switched to low level operations so comparing the fatigue life of its airframe to the V bombers isn't really fair.