Exactly. Determining whether a reference is "cumulative" to previously-cited art is too great of an unknown and the consequences of the applicant making that determination incorrectly are too great. The risk of having the patent found un-enforceable is huge, and it's most simply mitigated by citing the reference.

Long IDSs are the logical result of Rule 56 in its current form. It comes down to the balance between the public's interest in the applicant's full disclosure with the applicant's interest in not having his patent found un-enforceable when he didn't really do anything malicious. Right now, the balance on this matter is tipped more in favor of the public, and less in favor of patentees. Recall that a patent can be found un-enforceable for inequitable conduct even if a withheld reference does not defeat patentability. The question is merely whether it's something the examiner would have found "material" to patentability.