As a disclaimer, my undergraduate degree and PhD were in chemistry, rather than physics, and in the UK, not the US.
When I started my PhD I was planning on staying in academia. By the end of it I was desperate to leave it behind forever. Organic chemistry is somewhat notorious for having some very strange ideas about what constitutes an acceptable work/life balance. It's generally accepted (and emphasised most strongly by the more successful and/or ambitious groups) that as a PhD student or a post-doc, your work is your life. Six days a week is standard, and if you're not still in the lab by at least 7 o'clock in the evening then you're a slacker. As an aside, this leads to extremely poor time management practices, since the accepted solution to any perceived problem is "throw more lab hours at it"; this is partially due to the nature of the field and organic chemistry still being a touch unpredictable and requiring large amounts of experimental work to offset this, but it's an endemic part of the working culture. It also leads to people being in the lab just to be seen to be in the lab, rather than using their time productively. It's ridiculous.
There was a study commissioned by the Royal Society of Chemistry a few years back looking at why chemistry had such a poor retention rate of women. Physics has a low proportion of female academics, too, but then it has a relatively low proportion of female undergraduate students. Chemistry, on the other hand, has roughly equal male and female intake at undergraduate level, but the further up the ladder you go the further the ratio becomes skewed in favour of men. So what's up with chemistry? The conclusion was that the field fosters tribal attitudes to adversity (your PhD is a trial by fire!) and very masculine support systems, and that long term prospects are not very conducive to family life. I remember reading a related quote from a US professor which, to paraphrase from memory, said: "I can give you a list right now of all my former [chemistry] students who had a good handle on their career prospects. They're in my 'recommendation letters to medical schools' folder."
Funding is short for post-doc places and shorter for academics. But there's always industry jobs, right? Wrong. The jobs barely exist. Where they do exist, they're poorly paid, unstable and have poor promotion prospects. Anecdotally, when I was looking for jobs at the end of my PhD the going rate for an organic chemistry industry job (post doc experience preferred) was around £22-24k. That's less than what a sociology student going for any of the generic graduate schemes at a thousand different companies can expect to get straight out of their undergraduate degree, and with less opportunities for advancement to boot.
So if you want to have a life outside of your work, pursue hobbies or outside interests, start a family, buy a house, be relatively financially comfortable - a career in chemistry (I won't generalise to "science", that would be overreaching) is a very, very poor choice. It won't change, either, because there will always be someone who will be willing to work 12 hour days 6-7 days a week for the prospect of just one more publication. Is it worth it? That's obviously up for individuals to decide, but depressingly enough the smartest thing I could have done with 9 years of scientific training at world class research institutes was to use it as a springboard to get the hell out.
I'm much happier now.