One of the errors I found is that on your else block, mat2 is not defined. Because of that if random numbers take me to the else block in first iteration, the code error :
Error in mat2[rnum] <- 1 : object 'mat2' not found
The error that you have mentioned is occurring because the while loop is only a statement, there is no enclosing function in your code and hence the return(mat2) statement is incorrect. You can only use return if you want to return from a function. In your case there is no function and hence the error