What's the difference between consistency and extremism?

I'm often called an "extremist" for my views -- in my view, because I'm very consistent and refuse to compromise. Religious people are often called extremists too, yet that's really only consistency with their scripture. So how does "extremism" differ from consistency, if at all?