You want to respond to long-term (low frequency) changes in price, but ignore small short-term (high frequency) fluctuations. So what you need is a low-pass filter.
There are lots of different kinds of low-pass filters with different characteristics. A moving average is one kind of low-pass filter, but it has the problem (which you observe) that it trades smoothing for lag (in order to smooth out high-frequency fluctuations you need to average over quite a long sequence, but then it takes a long time for low-frequency changes to become apparent).
So you might try out the other low-pass filters, for example the exponentially-weighted moving average, and see if they meet your requirements.
The other thing to say about this task is that basing your decision about when you do a recalculation on the nearest integer to your moving average is a bad thing to do, because it amplifies small fluctuations when the moving average is close to halfway between two integers. (If the moving average goes from 99.49 to 99.51 the rounding operation amplifies this small fluctuation to a big one from 99 to 100.)
What you should do instead is use hysteresis to avoid this kind of response. When you do a recalculation, record the current value a0 of your moving average a, and don't do another recalculation until the moving average a has moved by at least ε, that is, |a − a0| ≥ ε.