Instead of IsMemberOf
method you should use GetMembers(Boolean)
with 'true'. It will return all the members of the group - even nested. Then make a loop to check if your user principle is in the result. Check this link.
Additional note: try such code
public static bool IsUserInGroup(string dc, string User, string group)
{
bool found = false;
PrincipalContext ctx = new PrincipalContext(ContextType.Domain, dc);
GroupPrincipal p = GroupPrincipal.FindByIdentity(ctx, group);
UserPrincipal u = UserPrincipal.FindByIdentity(ctx, IdentityType.SamAccountName, User);
found = p.GetMembers(true).Contains(u);
p.Dispose();
u.Dispose();
return found;
}