Are egoism and altruism mutually exclusive?

Most people have a common-sense view of ethics. They think that a person should spend lots of time pursuing his own goals and happiness. They also think that a person should sometimes set aside such pursuits to help others. Basically, on this view, a person can be an egoist and an altruist. Yet I've heard that egoism and altruism are two wholly incompatible moral theories too. So what's right or wrong about the common sense view?