I don't see anyone giving actual, useful advice here. I face similar problems with Amanda, and so far my solutions have been to lower the barriers to entry and to encourage users to increase their investment in supporting the product.
I've tried to lower barriers by
- writing better developer-level documentation
- providing a list of "starter projects"
- giving talks and webinars about Amanda's internals
- rewriting parts of the application in a more accessible language (Perl)
- making myself highly available for answers and advice
I've tried to increase investment by
- prominently displaying the names of contributors in ChangeLog, NEWS, etc.
- asking users to become "official" supporters (platform experts)
- requesting testing from specific people, rather than sending blanket "everyone please test this" emails
- asking users to donate processor cycles to automatic testing (this is still in the works)