We're all made of the same stuff.
Up until around six weeks, there's no difference between male and female embryos. At that point, a tiny region (usually) found on the Y chromosome called SRY activates. Less than 1000 base pairs, its job is to start the virilization cascade. It's highly mutable, so it tends to be prone to "breaking" or transferring between X and Y - leading to XX males and XY females. But XX males lack the azoospermia factor in the Y chromosome's long arm and XY females have streak gonads, so while this randomly happens, they're infertile and the mutation doesn't persist. But otherwise they're phenotypically normal men and women, up to the point of infertility in XX men and a lack of puberty (due to the nonfunctional streak gonads) in XY women.
At the start of the virilization cascade, everyone has the same basic set of organs, including the urogenital sinus, paramesonephric (Müllerian) ducts, and the mesonephric (Wolffian) duct. These are to form the common, female, and male organs, respectively. The paramesonephric ducts will develop if not exposed to anti-Müllerian hormone (AMH), and will fail to develop / degenerate if exposed to it. The mesonephric duct will develop if exposed to testosterone and fail to develop / degenerate if not exposed to it. Note that this is two different hormones - more on that mix-and-match later.
Most aspects of the genitals however come from the common urogenital sinus, leading to cognates in both males and females: labia-scrotum, clitoris-glans, prostate/paraurethral glands (as well as the lower 2/3rds of the vagina). This leads to a smooth interpolation between the two (diagram here). To reiterate, these pairs are the same organ, just grown to different shapes / sizes. A glans is a large clitoris. The scrotum is fused labia. Etc.
So we've already accumulated quite a list of things that can go wrong, including defective SRY, transferred SRY, unusual karotypes (X0, XXY, XYY, XXX, XXYY, etc), presence / absence of AMH without the absence / presence of testosterone, insensitivity to AMH / testosterone, etc. Using a very broad definition of intersex (e.g. including unusual karyotypes, such as Kleinfelter syndrome (XXY), up to 1,7% of the population deviates from the normal developmental process. For visibly ambiguous genitals, it's about 1 in 5500. A couple examples:
Androgen Insensitivity Syndrome (AIS): a largely (PAIS) or complete (CAIS) phenotypically normal female, but XY. Generally infertile. Exposed to androgens in the womb but don't react sufficiently or at all to them. Generally identify as female.
5-alpha reductase 2 deficiency: XY, but the body doesn't produce much / any 5aR2D, which converts testosterone into the more potent dihydrotestosterone (DHT). Born largely phenotypically female, but at puberty the testes descend, the voice deepens, the clitoris enlarges, and they undergo a relatively normal male puberty - leading to the nickname in the Dominican Republic (where it's most common) of "guevedoces" ("balls at twelve"). Despite being raised female, they typically identify as male, and - with medical assistance can sometimes father children.
Of course, in addition to primary sex characteristics you have secondary sex characteristics, developing at puberty due to whatever hormones the person is exposed to. E.g. one's larynx isn't taking a gander at what genitals one has - if it's exposed to testosterone, the voice will deepend, and if not it won't. Same with body hair, breast development, etc. E.g. male nipples aren't atavisms; they're just undeveloped tissues that never got the signal to develop. One is reminded of the scene in "Meet the Parents" where Greg, trying to impress his would-be father-in-law describes milking a cat, and says anything with nipples can be milked - to which the father in law replies, "I have nipples, Greg - can you milk me?" Except, yes, the answer to this is "yes" - expose his body to estrogen to develop the breasts, then to prolactin, and he'd lactate just like anyone else.
To loop back: we're all made of the same stuff.
