Should verbal or physical attacks on short people be tolerated?

Given your view that anti-gay bigots should be ostracized, what would you say about people who attack -- physically or verbally -- people who are shorter than most, say under five feet tall but otherwise perfectly normal and healthy? Many small people have long histories of physical, emotional, and even job-related abuse because of their size. How should those people respond to such attacks? What should third parties do when they witness such attacks? How should people judge and treat the attacker in future?