A DC offset is a frequency component at 0 Hz. The "wandering DC offset" will be made of very low frequency components, so you should be able to remove this by using a high-pass filter with a cutoff of around 15 Hz. That way, you'll remove any sub-sonic DC related stuff without altering the audible frequency range.
Use a filter with a steep rolloff. Seeing as you're doing this offline, you can use a simple IIR type and filter the signal in both forward and reverse directions to remove any phase distortion that would otherwise be imposed by the filtering.
If you use matlab, the operation would look something like this . .
[x, fs] = wavread('myfile.wav');
[b,a] = butter(8, 15/(fs/2), 'highpass');
y = filtfilt(b,a,x);