Comment Re:Trained on human written content... (Score 2) 80
It has been known for a long time that the behavior exhibited here is sub-optimal. Reputation effects have been shown to matter in games that are played for more than a single turn. An agent that backstabs on one turn cannot be trusted during future turns. Tit for tat, a strategy of cooperation, followed until another player backstabs and then punishing the backstabber, followed by an attempt at collaboration again is the strategy with the highest payout. In a long running game, that should eventually settle into cooperation for longer periods rather than what is described here which seems to be lots of unnecessary attempts to screw over the other guy. See, for example, The Evolution of Cooperation, by David Axelrod, 1984.
My PhD microeconomics classes in the mid 1980s had a hefty dose of game theory looking into these kinds of problems for oligopolistic markets where each agent used its observations of other agents' pricing to help set its own pricing. The result was that you could get prices set by each agent that looked collusive even though the agents had not explicitly talked to one another.
This shows an important distinction between LLMs and real profit maximizing agents. The LLMs don't seem to be able to learn over time what is the best long run objective function. Statistical token streams don't give rise to any real self-analysis where an agent asks itself if the strategy it is using is the best strategy or could be improved. I'll bet that a simple deterministic program that uses observations of other agents pricing coupled with tit for tat punishment and which never explicitly talks to competitors would beat programs like Claude hands down at a profit maximizing game.