Are the religious freedoms of Christian bakers being infringed?

Over the past year, the news has been inundated with stories about Christian bakery owners refusing to bake cakes for gay weddings. These bakers are defended by the Christian right for exercising their "religious liberty" and decried by the secular left for discrimination. I am somewhat sympathetic to the bakers, though I am gay myself. However, it isn't their religious liberty that's been violated, but their right to their property. When I speak to people about this issue, they don't understand where I'm coming from when I say the property rights of these bakers are important. Yes, their actions are motivated by a ridiculous moral code, but that isn't the issue. What is the best way to respond to people who think that these Christian bakers should be forced to bake cakes for gay weddings? Given that businesses are already prohibited by law from discriminating against other minorities, would it be so wrong for the law to encompass sexuality-based discrimination too?