How often should you change pairs?

Question

In the company I'm currently working on, we've been doing pair programming for a while now. In the current project, for the last couple of days, we've been trying to make our pair programming a little better, so we even have a timer now to ensure that the swap between driver and navigator happens often enough.

For changing pairs, we tried to do it in several ways. We tried changing at the end of the week, but found it too long. We tried in the middle of the week but we tended to forget, so we decided to do it daily.

Problem is that there are some stories that take a couple of days, so it's a little cumbersome to do it this way. Then we changed to a "per story" pair changing. But this is just a little uneven because the stories themselves most of the time are uneven as well.

So, simple question:

How often should you change pairs?