i'm currently learning and implementing td learning, https://en.wikipedia.org/wiki/Temporal_difference_learning it's a good read.

look into how our neurons give us the perception of time passing and you'll find your answers. traditionally, reinforcement learning doesn't have any temporal statistics and mostly just relies on the fire rate.