Both Yacoob and Afabbro have some great lists above (especially Afabbros list!). A combination of these features would be an ultimate system.
I work for a company called iQuate - I amd the CTO and have been (sometimes literally) developing a monitoring system for about 7 years. We do many of the things mentioned in the 2 lists, but not all (I wish!). We have a product called iQRMS which integrates several functions, the largest of which is monitoring.
- It is agentless - it uses about 30 different protocols (including SNMP obviously!) to connect to remote machines, so it can be deployed very quickly and gives a pretty "true" picture of client connectivity (which sometimes an agent based approach will not).
- It is horizontally scalable (you can have many scanning services on many computers and they will load balance between them).
- It has failover built in - when 1 or more of the scanning services die, the others redistribute the load.
- It has intelligent aggregation of data, recording max, min and average values for any monitor over time - for up to 6 years - in such a way that it doesn't just eat disk and kill performance (that one took a while to crack...)
- It has pretty graphs and in-depts reports on events
- It supports complex (or simple!) escalation rules to control who gets told about what, when and how often when events happen
- It integrates with a helpdesk (it's own or others)
- It allows you to create templates of monitors using different protocols to get a wider picture of an issue
- It is easy to understand and designed with 24x7 operations in mind (hence all that failover/scalability)
- It doesn't cost the earth
It also doesn't do some of (1 of) the things Timothy mentions at the start of the post (gratz on the new job btw!) - specifically it doesn't create a 2D map of the environment, although there are some plans to implement that in future. It treats and represents devices in the network as groups of hosts - it doesn't display them in relation to physical layout...
Maybe it's worth having a look at it Tim, I can certainly vouch for the support being excellent (but like I say above - I'm biased :))
JK