Could I contribute while mountain-biking? Could I contribute by ballroom dancing? Could I contribute while driving miniature steam engines in the park on Sundays? Could I contribute while acting in local Shakespeare plays? Could I contribute while woodworking? Could I contribute by going to the movies?
It is simple, most people have hobbies that they enjoy spending their spare time on.
Just because some people have a passion for Open Source and others find utility in it doesn't impart any sort of onus to assist development. Isn't that the ethos of Open Source - you can use it with no strings attached?
You might as well ask the opposite - Why are there so few FOSS coders just dropping in at rest homes to talk to the elderly? Why are no FOSS coders painting murals in public spaces? Why are no FOSS coders picking up rubbish in the park? Why are no FOSS coders building mountain bike trails in the weekend?