I mean the total universe, not just the observable universe.
The universe is 14b years old; let's assume it continues for another 14b years. Ignore expansion for now, and we can conclude that it must be at least 14b light-years across because we can see almost back to the big bang. In a 2D space-time diagram (1 space dim + 1 time) you'd see a square with an X in it - we're at the center, the absolute past is below us, the absolute future is above us, and the "neither past nor future" (let's call this the "elsewhen") is to the sides. Clearly half of this "universe" is in the elsewhen.
Now make it a 3D space-time diagram (2 space dims + 1 time): the diagram is a cylinder and the light-cones are normal geometric cones. But this means the volume of the absolute past + future is less than half of the cylinder, leaving more than half for the elsewhen. Add in the 3rd space dim, and there's an even smaller proportion inside the "hyper-cones" and thus even more outside.
Now if the universe if really more than 14b LY years across (which it almost certainly is - some estimates are hundreds of billions of LY, some much more), there's even more space-time in the elsewhen.
And because it's expanding, then the "hyper-cylinder" flares out as you move from past to future, resulting in yet still more space-time in the elsewhen.
So ultimately a very tiny fraction of all of space-time is in our absolute past, and a very time fraction is in our absolute future, meaning the vast majority is in the elsewhen.