I used to think the same, until I got video from my Samsung S4 and compared it to my Canon T2i. I can't tell the difference between video in a lot of cases.
Now, the T2i isn't primarily a video camera, but it has far better lenses and a far better/bigger sensor than the S4. It should perform substantially and irrefutably better than the S4. It doesn't.
As far as photos go, I'm either going to get out the big DSLR or I'm not. The DSLR obviously has superior image quality, but the camera is too big to carry around all the time, and is a significant theft risk if left unattended. If I have to wrangle 2 kids and whatever bags of stuff they require, I don't have a lot of patience for carrying around a camera bag too. Point-and-shoot consumer cameras are pretty much the same thing as cameraphones, unless you are talking about large models with big lenses (which have the same size problems as DSLRs). The cameraphone goes in my pocket. Of course it can be stolen but it is unlikely to be stolen if it is in my pocket. With the larger cameras you have to constantly be on guard and conscious of possible theft.
The best camera is the one you have with you. 1 device to rule them all is perhaps not the ideal solution but it is the most practical.
For the same price you could get a cheap feature phone, a tablet, and a camera. Use the phone for tethering the tablet, which works better for browsing the web and other such functions, and use the camera for taking pictures. That way you don't have to worry about your $600+ phone when you just want to go hiking or go for a bike ride where you don't really need internet connectivity anyway.
I have enough complexity in my life already without juggling 3 physical devices, managing all the interfaces (networking, file transfers, charging) between them, and upgrading/replacing them when needed.