There are multiple issues with your code that will probably make it operate differently to how you expect.
You test for operation values of 1 through 4. operation is assigned it's value from rand() % 4. This means that operation will only ever have values of 0 through 3.
Your do while loops all have the same flaw. They test for ans != ans whereas you should be testing for ans != ans1.
Fix these problems and you will get a bit further.
EDIT to give you a better hint
if(operation == 1){
printf("What is %d + %d ?", num1, num2);
scanf_s("%d",ans1);
ans = num1 + num2;
if(ans != ans1){
do{
printf("Incorrect! Try Again!");
scanf_s("%d", &ans1);
}while( ans != ans1);
}
printf("Correct!");
}
Edit to show use of srand
int ans;
int ans1;
srand((unsigned int)time(NULL)); //I've included your (unsigned int) cast.
int num1 = rand() % 10 + 2;
int num2 = rand() % 10;
int operation = rand() % 4;