A free enterprise system, as a complex adaptive system in itself, will always tend to converge toward the most 'efficient' or 'minimal error' surface in the ecosystem.
This is all well and good in pre-specified / full-information systems, where both the space of potentials and the utility surface are known. The real world isn't very much like that. Given your CAS and Sante Fe leanings, you've surely read Kauffman. Adaptive systems, under real-world constraints, are generally poised-criticality systems. i.e., they balance centralized and parallelized processes. This is, perhaps, what you mean by needing both positive and negative feedbacks, but it's quite a stretch to call such a system 'capitalism'.
Modellers have the best of intentions, and often produce valuable insights on limit case invariants, but there's a routine overreach when these conclusions are exported back to the real world from which the original enormous simplifications were derived. Setting out to model 'capitalism' or 'socialism' is a noble effort – but it's a mistake to then say that your model is capitalism or socialism. It's merely an example of how a system might behave under the constraints you've chosen as emblematic of those systems. In the framework of policy discussions, this a dangerous thing to do – because even if the limitations and technical details are known and clear to you, the modeller, the consumers of your work will falsely assume that your conclusions relate to 'capitalism' or 'socialism' in the world, which cannot be disentangled from politics, religion, failures (or adaptive heuristics, if you prefer) in human reasoning, chance resource inhomogeneities, noisy and incomplete information, inability to predict future innovations and uses (and consequently an increasing error in estimating the utility surface for increasing time horizons), etc., etc.
That said, communism in most of its theoretical forms is definitely not a better idea – and indeed, numerous socialist theorists, even prior to the rise of the USSR, warned as much. Communism was correctly predicted by socialist thinkers to lead to a red 'bureaucracy' that would utterly fail to bring about any of the advantages of socialist and collectivist societies. However, I'd quibble on the language of 'instability'. If anything, the biggest flaw in centralized schemes is too much stability – i.e., a grinding stagnation even in the face of changing circumstances, and a piling on of compounding errors and inefficiencies. To be sure, this is a recipe for disaster, but it's not really instability so much as hyper-stability, or a 'crystallization' (in Kauffman's language) – a shrinking volume of the accessible state-space with a growing energy barrier, so that the system is doomed to simply break or short-circuit when the external forcings inevitably change too much...