Ya no... that's not how it works at all.
It's called a Fiber Mux: http://en.wikipedia.org/wiki/Multiplexing
That's how things like DSLAMs work: One (or more for redundancy) fat pipe for backhaul, a router or switch in the box at the curb, and individual links carrying only each customers' data to the DSL modem at each customers' site.
Passive Optical Networks work like cable internet (and vaugely like the original party-line coaxial Ethernet): A pair of light frequencies (one outgoing, one incoming) connect the box at the curb, through attenuator/splitters, to each of a handful of sites. (The one I saw had an 8-bit hardware address and handled 250 subscribers per fiber.)
- The outgoing signal contains the traffic for all up-to-250 subscribers on a given link and the subscriber box rejects traffic for all but its own destination(s).
- The incoming traffic takes turns on the other light frequency. (Timing information is on the outgoing link and they run a link-level protocol to assign slots as requested when the subscribers' boxes have traffic, rather than a collision/retry protocol.)
Advantage is you need about half as many optic transcievers to implement it, while optical splitter/combiners are really cheap.
So, yes, it would be trivial to build a box that could listen to the fiber and tap your neighbors' downbound traffic. (You MIGHT be able to tap the upbound traffic from SOME of your neighbors, too, with a sufficiently sensitive optic receiver and if the fiber joints and splitter/combiners have enough discontinuity to reflect enough of their inbound light.)