Firefox's plugins are both it's greatest strength and it's greatest weakness. The "API" isn't really an API at all, it's just Javascript running in the browser process where it can hack about with the UI. It's extremely insecure and prone to conflicts, or breakage as the UI changes.
And with great power comes great responsibility.
Addons have nearly unlimited control over the browser, allowing them to do all sorts of amazing and useful things. Part of the price of this is a flexible framework -- using Javascript inside the browser's context instead of some limited DSL or something -- and another part is a more fragile connection to the user interface -- directly creating and manipulating XUL via the DOM -- which really isn't horribly fragile since they've pretty good about keeping element IDs and class names for a long time.
Security between addons isn't an issue, since they're intentionally not sandboxed from each other (and that wouldn't even make sense). Keeping them isolated from web pages is simpler, since that's already required for core browser functionality. The biggest issue is making sure addons themselves don't expose the user (such as Greasemonkey's unsafeWindow), but again, that comes at the expense of the power that addons can wield.
at the expense of requiring add-ons to be rewritten.
This would kill Firefox, so they will never do it, and I'm fine with that. We would undoubtedly get something worse than we have now (e.g., Chrome's limitations).