Could it also be related to poorer parents working more hours, thus having less time to be with the kids during their early years
There are probably dozens of measurable ways being poor affects the development of brains. But ultimately, it's happening because that's the way the people at the very top in terms of wealth and power want it. The last think they want is for poor people to suddenly become successful, and develop political power.
One of the side-effects of the New Deal and the labor movement in America is that a bunch of people who were on the shitty side of the street suddenly were able to take a big step up. GIs coming out of WW2 who would normally have gone to work in the coal mine or slaughter house suddenly had the means to go to college, buy a house, live a life of reasonable comfort. And do you know what they did with this sudden shift in circumstances? They developed political power. Their kids went from blue collar to white collar. And similar steps up during the pre and post-war period occurred for women and American blacks.
And this scared the bejeesus out of the elite. So, in the 70s, and peaking in the 80s, there came an effort to undo these advancements. The effort included an attack on New Deal programs like Social Security and the labor movement. The effort included the flooding of the inner city with crack cocaine (which we now know to have been run by our own government). The effort included the confluence of evangelical religion with supply-side capitalism by Billy Graham (who started his ministry with funds from the Chamber of Commerce).
The effort to put the genie back in the bottle continues to this day, and it's been quite successful. Economic disparity hasn't been this out of whack since the 19th century. Militarized policing, private prisons, an attack on public education are all fronts in this class war being waged by the elite. You want to see it in action, watch the political policies being pushed by governors of Kansas, Indiana, and many states throughout the South and in big cities in the North. Just watch.