You don't need apex for that?
<apex:inputFile>
has accept
parameter which you can use. Bear in mind this will check contentType, not extension (which is bit more proper way to do it).
If you still want the validation in apex - probably something like this?
String fileName = 'foobar.xls';
Set<String> acceptedExtensions = new Set<String> {'.doc','.txt', '.jpg'};
Boolean found = false;
for(String s : acceptedExtensions){
if(found = fileName.endsWith(s)){ // yes, there's only one "=", I do want assignment here
break;
}
}
if(!found){ // after the whole loop it's still false?
ApexPages.addMessage(...);
}