Should we stop caring about the sex lives of politicians?

In response to the affair and resignation of David Petraeus, many have argued that such sex scandals are the absurd consequence of American puritanism. These people claim that sex is easily compartmentalized in a person's life, such that sexual fidelity has no bearing on a person's intelligence, character, or suitability for public office. Is that right?

See: http://davidsimon.com/stray-penises-and-politicos/