It's too cold to hunt very often aside from big game that'll keep because of the cold which left plenty of time to "keep warm" so I can easily see this being accurate for our early ancestors too.

It's also possible they had a breeding season i mean most babies now are concieved between june and November as is. Then there's the fact that jealous males in other species kill any child they don't think is there's, this may also be a contributing factor.