My understanding is that we are currently living in an interglacial period.

I believe the pattern on Earth recently has been you get these relatively mild climates that last around 10 thousand years, in between 100 thousand years of ice age.

So, maybe the climate was harsh pretty much everywhere that wasn’t the equator for most of time our species has existed.