The city granted you a permit to build what you want on this land, but only for X years. You chose to build a house and now want to pretend that the permit wasn't temporary. Everyone will spend money on lawyers, but in the end, you had full knowledge of the situation when building the house.