Clock Tower for the PSX did this... You had no idea when Scissorman would show up. And if he did, you had two options: Run, or fight him. You had maybe a 1 in 4 chance of beating him in a straight-up fight. And even if you did "win", you'd just knock him over so you can run past him, and he'd be back on his feet in a few seconds. This led to an unexplainable fear of him in me. Yes, he was the silliest looking murderer ever, but what he could do to you was genuinely terrifying.
A short guy with a pair of huge garden shears doesn't sound scary; until you realize that he is, despite his appearance, very intelligent and skilled in combat. There is not a firearm or even melee weapon to be found in Clock Tower. All you can do is hope that you can trip him or kick him in the junk so you can get past him if you get stuck in a corner. And you can only do this three times before you get exhausted and get killed the fourth time you try.
If you took the "RUN!!!1" response, hiding from him wasn't exactly easy, either. You'd think that hiding in a bathroom stall would be enough; it isn't. In the time I spent with it, I'd usually get killed every time I hid, unless I put a significant amount of distance between the two of us before hiding. For example: I was being chased down a hallway, and stopped, because I couldn't decide if the stairs were a better idea than the bathroom. I went for the bathroom as he got closer and closer. Hiding in a stall, I waited. He came in, looked stumped, and walked around a bit. Then, he did something I hadn't thought of: He started forcing his scissors through each stall door. I died instantaneously with no chance of correcting my error. I never tried that again.
So, yes. Having enemies that are more than powerful enough to kill you, without you being able to fight back is terrifying. A non-horror game managed to pull this off, and I'm pretty sure that it wasn't their intent. Breakdown had non-human enemies that required a special power to kill, or even harm. Once you realize that every weapon in your arsenal is useless against them, and they can kill you in one or two hits, you experience pants-wetting fear of them. You're forced to crawl through fallen bookcases to get away from them, hoping they don't grab your leg and kill you quickly afterward. There's a segment where two of these guys are locked in separate rooms, divided by a glass door from each other, and a glass door between them and you. Your goal is the door behind them, a steel door opened by a wheel. Pressing the button next to the door, or inside of it, opens ALL of these doors. If you didn't think to trap them, you have to run for your life.
What makes something scary? The environment, the enemies, and your range of abilities. If any of these is out of balance, it ruins everything. Your death must be an easy thing for your enemies to achieve for a game to be truly scary, but you still have to be able to hold out long enough to stun the bigger guys or kill the smaller ones for it to still be playable. It's a fragile balance, sadly.