A huge amount of it is in the presentation, and how the player conceptualizes the puzzle, and how much of the problem can be handled automatically by visual processes.
It's not just that. The puzzle solvers essentially adopt their own rules for what consistutes a valid solution step.
Once I started completing most five stars puzzles in twelve minutes or less, I started to mainly work the "insane" category. My preferred method is to logically eliminate a single digit placement: this digit can't go here in this box. In the insane puzzles, one often gets to a place where are few digit placements one can reasonably crack with an inference chain without going more than three levels deep.
At that point, it's pretty easy to get completely stuck for ten minutes looking in all the wrong places. That same situation can often be "solved" in under a minute with a four colour pen and the willingness to posit and backtrack. Normally this is against my personal code.
Once upon a time I compiled Knuth's dancing links and threw some "hardest ever" puzzles at it that came up in a quick Google search. One of the first such puzzles I tried solved without a single backtrack step: at each point where the algorithm made a guess, it happened to guess right. There were only three or four or five such junctures of valence 2 or 3. I think I had slightly modified how it sorts the list based on my own intuition about the potency of guesswork, but still, it made a completely mockery of the whole "difficulty" notion. Purportedly one of the hardest of all puzzles (by a certain metric) and Dancing Links goes Rain Man without so much as scuffing its eraser.
When I challenged myself to solve five star puzzles in under ten minutes, there was a complex dance inside my mind to keep track of where I'd shaken the tree already, and what part of the tree needed to be revisited based on recently completed digits. At the slightly faster pace, my mistake production would skyrocket: somehow my double-checking circuit and my "what next" circuit became competitive.
Also, the critical junctures became too thin on the ground, and the punishment for my errors too great, so I lost interest in pushing it any further. It was always for me an excercise in observing my own solution strategies and mental capacities/incapacities.
I think the only way a puzzle-setter can get consistent solution times for a hardness category is by patiently training the puzzle solvers to appoach the task in a certain way, rather than just doing their own thing. I certainly knew with each puzzle setter that I could exploit my familiarity with previous puzzles set at that level of difficulty if I followed the main sequence.
From time to time I would spot an advanced inference early, and then the puzzle would melt away posing no further difficulty whatsoever.