It's quite simple. A gay couple walks into a bakery and orders a dozen cupcakes. They should be served like any other person walking into the bakery. A gay couple walks into a bakery and wants the baker to make a specialty cake, just for them. The baker should not have to comply with the order. Forcing the baker to comply is involuntary servitude.
As long as that baker does not do other specialty cakes I would agree. What he does in private is his business.
There is your answer. Exactly as the New Mexico court held. The baker makes the specialty cakes by private arrangement. Unless someone is in a position to be able to request a specialty cake, the baker can tell them to take a hike.