That's the premise of a Threshold-Based Secret Sharing solution. There are multiple ways to provide a scheme where you need some X of Y shares to recover a value based on WHO is trying to recover the secret.
The simplest is to implement a scheme where N shares are generated and M (where M is less than/equal) to N shares are required to recover a secret.
The same secret can be split into different numbers of shares (or, even splitting a particular share into more shares that get distributed).
Assuming that all the share holders don't meet on Facebook and collude to unlock your secret, you can comfortable distribute the shares and they need not be encrypted (they are useless without the quorum of shares).
Of course, all it takes is one or more share recipients to encrypt the share (or have it encrypted prior to sending it to them using asymmetric encryption) and then it will require those individuals to decrypt the share before your secret can be revealed. It may be, theoretically, harder to compromise an individual AND their share repository vs just the repository.