In a normal RTD server, you use the excel callback to call UpdateNotify(), and then Excel will decide when to call RefreshData() based on the throttle settings and how "busy" it currently is. The definition of "busy" normally includes things like formula recalculations, saving the sheet, and having a modal dialog open.
Excel protects itself from external components when it considers itself busy by throwing an exception. This makes sense, as they don't want their application becoming unstable or locking up because of a greedy addin or macro. Essentially all calls you put in your code which access any part of the Excel object model (including your call to RefreshData()) can throw an exception if Excel decides it is "busy". I suspect that in later versions of Excel they have tightened their definition of "busy", which is why you are seeing this disparity with the modal dialog.