Technically, they were likely giving to philanthropy, putting resources into neighborhood and state government securities and profiting abroad.
The majority of the Multi-millionaires who did not pay taxes to the Government presumably still needed to hand over a lump of their change, just not to the U.S. government.
