No, it really doesn't - and there's nothing you can do about it. Supreme Court justices are appointed for life.*to you
Some people have a higher baseline of ethics and do think it matters.
It's perfectly consistent to allow interracial marriage but not same-sex marriage. Case closed.