Someone in the comments explained it really well, but plants need nitrogen and phosphorus to grow. Usually those are in limited quantities, so growth is limited by the amount available. Fertilizer is designed to provide an abundance of nitrogen and phosphorus to plants, so if those high quantities end up in the water, algae growth isn’t limited and can grow more rapidly.