[H]ow bad would something have to be before I would challenge it? I have, on occasion taken stands against bigotry that were potentially dangerous to myself--telling my project manager, for example, that his racist remarks were wrong, and that I wouldn't stand for hearing them.
I let a lot of misogyny slide, because, hey, it's not worth it, I won't change any minds, and besides, I'm frankly too arrogant to feel like it affects me.
At some point, like "I think we should herd all the Jews into camps and then kill them", you have to speak up.
Back in high school, and into my first years of college, I was in full-time combat mode. If I heard anyone say anything I disagreed with, I challenged him on the spot. Considering how appalling most people's beliefs are, it was a full-time job.
Over time, however, I've drastically changed my strategy. Now I only argue with people I respect, or for the benefit of an audience. (Thus, if we've privately argued, you now know my true feelings!) I don't respond to email from Communists, Nazis, and the like - not even to tell them, in the words of Tsunami Bomb, that "Talking to you is not worth my time."
On the other hand, if you're dealing with someone you respect, it's a tragic mistake to let them know, in Jane's words, that you "won't stand for hearing it." If you take offense, you make people afraid of offending you. And if you make people afraid of offending you, the quality of conversation plummets.
So what should you do if people you respect start talking crazy? Resist the temptation to use anything harsher than friendly ridicule. You'll still get your point across, and you won't kill the goose that lays the most precious of all golden eggs: uninhibited conversation.