Why are churches tax-exempt? Why do they get a free ride on our backs just because they worship God/whomever? WELFARE handouts undeserved?

Churches are tax exempt under the principle that there is no surer way to destroy the free exercise of religion than to tax it. 

Not being required to pay taxes is not receiving welfare.  That requires the assumption that your money, my money and organizations money belongs to the government and they allow us to keep some of it. I fundamentally disagree. 

Churches are not the only tax exempt organizations​. There are Unions,  other charities (Red Cross, PETA, PACs etc) are they receiving welfare?  If we change it for churches then we must change it for them all.