Not stupid at all. Single family zoning was explicitly created as a way around a supreme court decision (Buchanan v. Warley) that barred explicit racial covenants.

And in practice, Oregon has already banned it in most of the state, Washington is set to follow, and California has functionally done so in much of the state. Minneapolis has banned it city wide. Cambridge MA is generally expected to do so in the next year or two.