The anonymous function is the "user defined" or "custom" function you are after.
You just need to use the anonymous function according to your needs. For example, in order to consider only the 4 nearest neighbors (up down left right) pixels for a 3x3 window you can write:
f = @(x) max(x(2:2:8)) - min(x(2:2:8))
G = nlfilter(I,[3 3],f);
as the gradient.
Another example, to do the same for a generic circular neighborhood of radius r
:
r = 5;
H = fspecial('disk',r);
f = @(x) max(x(H>0))-min(x(H>0));
G = nlfilter(I,size(H),f);