Who or what defines rights? Who determined that a community can't establish the rights of its members and government institutions? If you answer that God provides us with our rights. What deity are you appealing to? The Christian God?
"God given rights" has nothing to do with religion. They're just talking about freedom, free will. We're born with the capacity for volition, and, absent of any interference, we have the freedom to act. The point of the phrase is to point out that we have freedom without the government. It's not granting it too us. Jefferson was trying to say that we create government to protect our freedom. Not to tell everyone how to live.