Solved!
My zoom calculation is based on an assumption that the X axis width in pixels is fixed and apparently when the InnerPlotPosition property is set to TRUE, the X Axis width in pixel may change during runtime, for example while zooming.
So the solution is to set the InnerPlotPosition property to FALSE and set the inner plot area manually:
chart1.ChartAreas[0].InnerPlotPosition.Auto = false;
chart1.ChartAreas[0].InnerPlotPosition.Height = 80.29209F;
chart1.ChartAreas[0].InnerPlotPosition.Width = 85F;
chart1.ChartAreas[0].InnerPlotPosition.X = 15F;
chart1.ChartAreas[0].InnerPlotPosition.Y = 5.07246F;