To add to the clarification by /u/PyrrhoTheSkeptic that what you're describing is the absence of heat in the surrounding soil (and thus heat within your basement is "flowing" into the surrounding soil via conduction), neither soil or rock are great heat conductors (i.e., they generally have low thermal conductivity). What this means, is that it takes a while for the temperature of the soil/rock at even a shallow depth to change after a change in surface temperature. Observations suggest that if, for example, you consider either diurnal (i.e., day-night) or seasonal oscillations in temperature, the amplitude of these oscillations decrease exponentially with depth (e.g., Elias et al., 2004). In other words, even though the air temperature may be warm or cold (and oscillate between them), the soil temperature at a few meters down will be more constant. You can see in data of very shallow soil temperature (e.g., Holmes et al., 2008) that you do see things like diurnal temperature variation in the upper most few cm, but within >15-20 cm, the magnitude of diurnal temperature variations of the soil are extremely small.

It's worth noting that if you go deep enough (few 10s of meters) that the temperature of rocks ceases to be influenced by either diurnal or seasonal surface temperature variations and is instead controlled by the local geothermal gradient.