Shouldn't egoism be limited to protect others from harm?

You advocate an egoistic ethics, according to which people should always act in their self-interest. However, isn't such egoism dangerous if unchecked? Wouldn't the egoist cheat, manipulate, and even murder others in order to get what he wants? Doesn't egoism need to be restrained by concern for other people and their rights?