I like this discussion very much. And yes, there are some theories that explain the defensiveness - see Milton Bennett on Intercultural Sensitivity for example.

My sense is that privilege is a conversation stopper because most people don't want to confront unearned privilege. The idea of unearned privilege does not imply the propriety or impropriety of earned privilege. I am often criticized for exploring class because I am not ready to confront ethnicity and gender, and my response is to turn that back on people and suggest that class is what they are afraid of discussing.

Will Barratt - one of the authors of the "Privilege meme"