I would agree with that theory, yes. It's pretty much a matter of empathy. Even if a character is evil, there are some traits you can identify with and respect. But with some characters... it's just impossible.
Same reason why most kinds of NTR have never bothered me, actually. The "victims" in these works tend to be so comically weak and dumb I don't really identify with their struggle at all, and so don't really feel bad for them (if you're getting into terrible shit, and instead of going to the police/get help you immediately agree to become X's slave... wtf, what's wrong with your head).