This has been going on for a long time. If one stretched things, they could say that World of Warcraft is a FPS with an extreme number of powerups. However, FPS is a concept is a known quantity. You put out something in this type of genre, and you will almost certainly break even at the minimum.
Finding new ways to do a RPG combat system is hard. There are not that many ways to do combat, so FPS mechanics is one of the most used. Turn by turn combat used to be the RPG mainstay, but for many people, it is too slow a method of resolving conflict. There are other combat systems, but if a game relies too much on arcade reflexes, it might turn people away.
For single player, most likely the best bet for a modern RPG these days would be a system used by NWN and NWN2, where people can pause the action before making their next decision, but if they know what exactly is going on, can still do an almost real-time battle.
What I've not seen that much of are RPGs with RTS mechanics. Picture having your group of people that you start out with at a beginning of the game, and each of them has some ability and weakness. There would have to be more plot and character development for an RPG to separate it from Warcraft 1-3 (adding multiple endings, having side quests), but it could be done.
One scenario using these type of mechanics could be pushing back some orcs [1] who are pillaging some nearby villages. You send in some scouts to see what exactly their weaknesses are (one village has an orc wizard fireballing buildings. Another has an orc chieftain who keeps his band up with heals. Still another has a warrior chieftain.) Then you send whatever guys you have that would be the best against the type of enemy at hand. With different playstyles, one could have a lot of grunt troops and just swarm the villages, send in ranged troops (with some melee in front as a distraction), or perhaps even find a way to use some type of negotiating skill to get the orc tribal leaders to accept a keg of ogre swill as treasure enough so they stop their invasion.
Another scenario could be a castle siege. You have your forces and need to punch a hole in the castle walls, while fending off forces coming from other sides. Part of the RPG would be doing side quests. One side quest earns you better siege engines. Another gets enemy troops to not join in on the fight. Still another side quest just might allow the player to earn such a famous/infamous reputation that they can just bypass the siege altogether and have the opposing side open the doors and surrender.
This isn't to say this has not been done before, but RPG/RTS mechanics are not something seen often in modern games. What sets RPGs apart from "plain old" RTS/FPS games is having multiple endings, multiple side quests, and different consequences for player actions. For example, if a PC is an extremely good diplomat, it may allow for some battles to be skipped or handled in a different way. Similar if a PC does side quests for a reputation. Throwing in some mini-games [2] may be the answer here as a way to help (perhaps use the RTS engine so the player can work as a mercenary general in order to help your side get land or resources in between plot advances.)
[1]: Classic AD&D/LoTR orcs which would be more than happy to stuff any intelligent race in a stewpot. Except dwarves. They are just too hard to clean.
[2]: One recent mini-game I liked was the Risk-like one in James Cameron's "Avatar". It was fairly tough because you had very little territory at the start, so you could either play your chances slowly, or start the mini game every so often, because you got more money as the main game progressed. Mini games have to be done right though. For example, the card one in FF8 a lot of players just skipped for the most part.