Time could be freezing for billions of years every five minutes and we would have no idea.

Edit: okay, due to time pretty much being the universe, we will pretend that the observer exists outside of it.