On a practical level, yes it is -- the microwave background stands in our way. As the universe expands it cools down (same as if you pump up a tyre both the pump and the tyre get hot, but in reverse) -- which means that tracking it back, in the past it was seriously fucking hot. The universe is also, even at the present day, composed of more or less 75% hydrogen and 25% helium. The ground state of the lowest energy level of helium isn't really very high, while hydrogen's is high but not *that* high. And if you start working it out, it turns out that if you look back to when the universe was very roughly 300,000 years old, you suddenly find a universe that was at the temperature where every single photon was energetic enough to ionise hydrogen (let alone helium). That then immediately implies that any photon would propagate a short distance (and a very short distance - the universe was vastly smaller then than it is now) before it was absorbed by a hydrogen atom which then spat out its electron. That electron would propagate a short distance before it fell into a proton and emitted another photon, which would almost immediately slam into another hydrogen atom, and so on.
That means that the universe was totally opaque. Using light, and we have no other probe right now (though a direct observation of a gravitational wave background would ease this somewhat, as would a brutally unfeasible detection of a cosmic neutrino background), we therefore have a hard limit back at the CMB.
Even if the CMB somehow wasn't there, yes, looking back we run eventually and inevitably into the beginning of the universe. The distinction is born entirely of the theory we're couching cosmology in -- general relativity or, at least, a geometric theory of gravity very similar to general relativity. In these theories the universe is actually described as a whacking great four-dimensional blob which we've sliced for convenience into spatial surfaces along some time direction. (Those choices aren't arbitrary; there are conditions on what you can choose as a time coordinate, and on what you can choose as a spatial coordinate.) It's those spatial surfaces that seem likely to be infinite in extent.
However, since we're working in something like GR we also have the restriction that we can't see outside of our past light cone. Nothing can propagate faster than the speed of light, and in GR that is actually described by the type of paths that things can propagate along, with light propagating along "null" paths, normal matter along "timelike" paths and either nothing or "tachyons" propagating along "spacelike" paths. These paths, intrinsically, *cannot* overlap. A timelike path will never be a null path and cannot cross one to become a spacelike path. That would be geometrically nonsensical. Spacetime is then mapped out by these "null geodesics", and if you map them back into the past you get what's known as a light cone -- formed of all the light that could possibly have reached us. What we can possibly observe has to come from within, or on the boundaries of, that light cone.
Now, that light cone *is* finite, and if we extend it all the way back to the singularity then we'll obviously run into problems. At a singularity everything genuinely dies, and our theory doesn't even begin to work. Even geometry doesn't. All this tells us is that our dynamical theory is missing something (which many - including myself - believe is a quantum theory of gravity that smooths out that singularity, quite possibly by enforcing a maximum density within a minimum volume, or some similar process). If we decide we're not going to wilfully put a load of infinities in our denominators and instead cut out light cone shortly after this singularity -- let's say when the universe was less than a femtosecond old; I'm sure that's young enough to satisfy everyone -- then we can even calculate the 4-volume of that light cone. And it is finite.
I think this might be one of the causes of the confusion on this point.
I'd also like to apologise again if I cam across as a bit curt.