I have a recent PhD in neural computation, though from a functional cognitive and language modeling perspective, and not a neuroanotomical modeling perspective -- so it may be a different area than you're interested in.
From a high-level perspective, neural computation has moved a lot in terms of scale in the past two decades (simulations can have millions of nodes), and it has moved a lot in terms of modeling the processes of individual neurons and neurochemistry. Very high-level functional mapping work has also moved a good deal with fMRI, EEG, and MEG becoming relatively inexpensive and very common techniques in cognitive experiments.
One area that, in my opinion, has moved very little in the past 20 years is the ability for neural networks to learn non-trivial domain-general representations and processes, and to generalize from those representations and processes to novel (untrained) instances. In the late 80s, after connectionism had made return with Rummelhart and McClelland's popularization of the backpropagation algorithm and demonstration of its utility in a number of tasks earlier in the decade, a good deal of the literature demonstrated very basic limitations and failures of these systems to generalize to untrained instances, or to move away from toy problems. Fodor and Pylyshyn's "Connectionism and Cognitive Architecture" is a classic paper from that era, and Pinker wrote a lot language-specific criticisms as well. Stefan Frank has the most recent long-standing research program in this area that I'm aware of, and his earlier papers have good literature reviews that can further help guide ones background reading. There have been some limited demonstrations of systematicity with different architectures (like echo state networks), and comparatively little work on storing representations and processes simultaneously in a network, but so far these are long-standing and fundamental issues that need revitalization. When convincing demonstrations do arise, they'll likely not need more than a desktop to run, as it will be demonstrations in learning algorithms and architectures, not scale.
For non-neural folks, classical neural network architectures are essentially very good at pattern matching and classification (e.g. being trained on handwriting, and trying to classify each letter as one of a set of known letters (A-Z) that it's seen many hundreds of instances of before), or things that involve a large set of specific rules (if X then Y). They're much less good at things that involve domain-general computation, that involve learning both representations and processes and storing them in the same system (i.e. let's read this paragraph and summarize it, or answer a question, or let's write a sentence describing a simple scene that I'm seeing). That's not to say that you couldn't make a neural system that did this -- you could sit down and hard-code an architecture that looked something like a von-neumann CPU architecture and program it to play chess or be a word processor, if you really wanted, but the idea is developing a learning algorithm that, by virtue of exposure to the world, will craft the architecture as such. The idea being that, after years of exposure, the world will progressively "program" the computational/representational substrate that is the brain to recognize objects, concepts, words, put them together into simple event representations, and do simple reasoning with them, much like an infant.
I hope that helps. Of course all of this is written by someone interested in developmental knowledge representation and language processing, so it may be a completely different question than you'd wanted answered. best wishes.