Consider the people you most admire that you know personally, such as your parents, spouse, or work mentor. Now imagine the worse [sic] sort of things someone might plausibly accuse those people of. Are you confident you really want to know if such accusations are true? If yes, why don't you look them in the eye and ask them point blank, just to lay the issue to rest? Or offer to bet them on it? I didn't think so.
At the outset, note that Robin limits himself to "plausible" accusations. Since the really bad stuff is rarely plausible - Robin's highly unlikely to be a serial killer - most scary accusations aren't worth making.
Furthermore, even for plausible accusations, a truth-seeker still might not ask, because: (a) If the accusation were true, it would be denied, and (b) people resent being accused. Just because you value truth, doesn't mean you don't value friends.
In short, the fact that we rarely accuse our favorite people is weak evidence that we aren't truth-seekers. It could just be a sign that accusations damage relationships without revealing much truth.