It's not only about perceived safety from driver abuse, safe driving etc. Regulations are there -- at least in some european countries, like Germany for example -- to ensure a standard of operating safetey, both from technical and commercial point of view.
For example, regulated taxis have stricter requirements with regards to technical maintenance. This is something you generally want! Just think about that for a moment: every time you take a ride, you're otherwise getting into some stranger's car, which could have been checked as long as 23 months ago for technical flaws, and may have been driven for 200.000 km or more during that time (he's a professional driver, being on the streets for a living). May have working safety belts or not, may have working airbags or not, bad tires... Hell, it may not even have properly working brakes for all you know! Not good.
Authorities actually go a long way to ensure that taxis adhere to more reliable technical standards.
Then there is the insurance issue: regardless of how safe, sooner or later an accident will happen. May not be with you in the car, but somebody else... but it's still happening. In that case, you want to be certain the driver is properly insured, and his insurance policy will cover any damage that may occur to you (life-long wheelchair for example?).
This is something that a taxi licence in civilized countries will ensure. Not happening outside the regulations, though.
Want to drive people around? Fine. Check your car as often as required by regulations, buy the proper insurances, respect any other passenger safety measures *and* *document* *that* *process*, then you're good to go. Oh... that costs money? Can't compete with taxis then? ...well, guess why!...