Take a look at RapidInsight It is an open source integration, automation and presentation solution for IT management.
Transparent Planned Design
Many solutions out there seems to have been developed in what can only be described as an "organic" process. I.E. a few scripts were used from start, were hooked up with some other scripts, were slammed into a web-interface, got some more features, then something central were ripped out and replaced to allow yet more features and so on and so forth. (Read: Nagios) While this is of course often the best way to get something working for a particular need, and on a tight budget, it makes adoption really hard unless you happen to have exactly the same need.
RapidInsight is developed as a platform first. Applications from data model to UI are built using the platform, hence matches to your "Transparent Planned Definition". The platform is using other common open source projects and tools wherever possible (groovy, grails, compass/lucene, etc.) to make it as easy as possible for others to modify/enhance the applications provided and build new ones. Current version of RapidInsight is built on the third iteration of the platform.
Event management
Does anyone know a solution that can both receive from syslog and decode traps with a given MIB, and then do some simple logic, like squashing repeats, displaying on a web-page with archival-options, and dispatch to mail/sms based on configurable rules? Except for ZenOSS (and ZenOSS have other problems), I haven't found a single sensible system that does this.
RapidInsight provides robust event management capabilities; enrichment, filtering, automated action, event lifecycle, archiving, full text search etc. RapidInsight can process syslog files and SNMP traps, provides XML/HTTP api, and has plugins for other open source (Hyperic, OpenNMS) and proprietary (Netcool, Smarts) managements systems. It provides a web based user interface (Ajax) that is capable of handling very large data sets. Events can be manipulated using groovy scripting language.
Modularity/Seamless Integration
Since much of the monitoring systems out there doesn't seem to have a clear design, it's often very hard to add missing features. I.E. project X missing an event manager, or is the builtin not satisfactory? No probs, I'll just, ehh, where does this wire come from? Is this really a socket? Did anyone really connect that? It's ok with blackbox-solutions, as long as they serve all my needs, and have clear interfaces to combine with other solutions that serves related needs, but sadly no solution evaluated does everything we need it to and we end up struggling with manual routines to compensate for it.
Seamless integration was a primary design requirement for RapidInsight which reflects on many aspects of the solution. For example, the integration layers abstracts the differences and provides consistent methods to work with external systems, databases, etc. Available applications all use the integration services provided by the solution.
Complexity
There are a few really neat systems that does almost everything one can ask for. (Short of flying cars). Unfortunately, the ones we've tried have always turned out to be very complex, and also do a lot of things we didn't want. Since it's then often not very modular, it hard to get it stop doing the things we don't want, or change the things we need implemented slightly differently. Also the huge codebase that comes along with trying to scratch everyones itch seems to get it's share of bugs, and troubleshooting in large more or less opaque systems is not a fun task.
This is a tough one. You're right trying to scratch everyones itch can drive a solution towards complexity. We struggle with this all the time. I'm too close to the solution to tell how well/badly we've done from this perspective. One good thing is that RapidInsight is modular which can reduce the complexity.
The Perfect Monitoring System
After evaluating all options we could find, we've come to the conclusion that none of the systems we've looked at or tested really fits our needs (Although ZenOSS came close, we encountered just too many bugs and oddities to keep investing time in it). Furthermore, we could not find a combination of systems that integrates well, and together fits our needs, which I personally see as a bigger problem.
What I would really want to see in the world of Open Source Monitoring, is an eco-system of monitoring apps with an overarching design/architecture. Design a framework where different entities and steps in the monitoring are clearly defined and interfaced with each other, but still allows for differing implementations, and integration with unforeseen needs. For example, at our shop, we continuously analyze roughly 700mbit of streaming video for availability and quality. Noone designing a monitoring system could probably forsee this as an appliance, but in The Perfect Monitoring System, it should be clear for the average-skilled hacker how to integrate it.
Perfect monitoring system is hard to achieve, not to mention different people have different definitions of what's the "perfect" solution. While designing RapidInsight, we have accepted that it is not likely/feasible/possible for a single solution to provide all the functionality required by diverse groups. Hence, RapidInsight is an attempt to integrate various tools used for different management disciplines and silos to provide a consistent and simple interface to users.
My sister opened a computer store in Hawaii. She sells C shells down by the seashore.
