Does ethics require impartiality?

Critics of egoism, particularly utilitarians, accuse egoists of being biased in favor of oneself without justification. They assert that a scientific ethics must be neutral and impartial: it must take a third-person viewpoint where the self isn't given any special consideration. Are the utilitarians wrong? If so, why should a scientific ethics bias the self over others?