You are defining the check
constraint inline and moreover subqueries are not allowed. You need to define the constraint at table level like below
create table UsersFriends
(
UserID int references Users(UserID),
FriendID int references Users(UserID),
primary key(UserID,FriendID),
CONSTRAINT CK_nested_friend check(UserID <> FriendID)
);
In case need a demo, see here http://sqlfiddle.com/#!3/1e405