Because maintaining massive stone structures is expensive, and often nobody owns these buildings, and lacks the means to make the upkeep a paying proposition. When times are bad, and money scarce, people's romantic attachment to history very quickly goes by the board.