The problem is threefold.
Firstly lack of updates, SoC vendors are notorious for porting one or two versions of Linux, throwing it over the wall to device vendors and then doing nothing to keep it up to date. Some SoCs can be use with upstream kernels but very often with reduced functionality. The device vendors in turn add their own customisations to that kernel that the SoC vendor threw over the wall. Quickly you end up with something that cannot reasonablly be updated to a new upstream version. It is possible to some extent to backport security fixes, but it's a lot of work so it is likely to get skipped entirely or at least restricted to the most-severe vulnerabilties.
Secondly the vendors doing the work often do it without really caring about security which can lead to busting big holes in the user-security model. Remember "exynos-mem"?
Thirdly if your application layer is full of holes then attackers will be able to get whatever privilages that application has. If that is root then the attacker has full control of the device. Even if it is not root the attacker may well be able to elavate to root due to the first and second points.