It is moral to advocate for the boycott of a business?

Over the holidays, my brother and I discussed cases in which businesses are compelled by government to provide services against their will. For example, the Colorado courts demanded that a bakery make cakes for gay couples or face fines. We agreed that the business should be left free to operate as they see fit, absent violating anyone's actual rights, and reap the rewards or penalties from their choice. Where we diverged was on the moral status of the business owner and whether the bakery deserved to be boycotted. In my view, the decision of the owner of the Colorado bakery was immoral: they were being irrational, discriminating by non-essentials. My brother disagreed. Moreover, my brother opposed any advocacy of a boycott, seeing this as a call for force to be applied against the owner. This would be wrong, in his view, but he would be fine with suggesting that people patronize a different store. Ultimately, I found that I could not adequately explain why I think people might actively and openly oppose wrong acts by businesses, even if those acts don't violate rights. So what justifies such boycotts, if anything?