Top reason I'm not an anime fan: time. Being an anime fan is a bit like being a "book fan": you actually need to watch a lot of it. I don't watch as much anime as I read certain types of book, watch certain types of films, listen to certain types of music, and I've no connection with the community.

There are some things in anime which casual watchers (like me) tend to find irritating, particularly fanservice and endlessly recurring character types. But I've come to love other types of media that arguably have at least equally offensive traits, so I don't think it's fodder for prejudice against anime fans per se ... that said many anime fans, particularly those who have little outside-anime literary context for their consumption, do tend to shit me, because they squee constantly about things that just aren't that mind-blowing.