I think that one thing entirely missing from the debate is the need for basic "drives" in an intelligent system. We define "intelligence" as the capacity to model complexity in the world.. but the world has infinite complexity, and any finite system needs to establish priorities about WHAT to model.
In nature, of course animals use their intelligence to satisfy their basic nees. All animals want to: breathe, drink, eat, breed, avoid being hurt / eaten.
More complex animals live in societies, and may also want to: nurture their offspring, choose the best partner, rise in the social hierarchy, etc. But these drives ultimately lead to the same basic goals.
Some neurotransmitters have been mentioned. Do you know their functions? Many of them drive the animal's behavior. For example, dopamine is thought to signal unpredicted rewards or unpleasant stimuli, so it is directly involved in motivating the animal. Oxytocin is thought to drive prosocial behavior: maternal behavior, love, empathy, etc.
My question is: what neurotransmitters do we want to "model" in an intelligent system? Why? Or in other words, what do we want the intelligent system to want? Do we want it to just answer our queries? If so, how would the system know what to model in order to be useful to us? Wouldn't it be required to also know what we want? How would it know what "want" really means?