I tried the same functionality of Matlab, and got NaN values. I dig in Matlab documentation and found the following which explains the NaN values:
"For higher order filters (possibly starting as low as order 8), numerical problems due to roundoff errors may occur when forming the transfer function using the [b,a] syntax." Therefor the butterworth filter is unstable for higher orders.
To get over this problem in Matlab, try the following:
n = 6; Wn = [2.5e6 29e6]/500e6;
ftype = 'bandpass';
% Transfer Function design
[b,a] = butter(n,Wn,ftype);
h1=dfilt.df2(b,a); % This is an unstable filter.
% Zero-Pole-Gain design
[z, p, k] = butter(n,Wn,ftype);
[sos,g]=zp2sos(z,p,k);
h2=dfilt.df2sos(sos,g);
% Plot and compare the results
hfvt=fvtool(h1,h2,'FrequencyScale','log');
legend(hfvt,'TF Design','ZPK Design')