I managed to solve this problem. Hope it helps someone in the future.
The problem was with the sequence in which I was adding the validators to the file elemenet.
Changing the sequence to below solved the problem.
$file = $this->createElement('file', 'templateFile');
$file->setAttrib('id', 'templateFile')
->setAttrib('class', 'form-control')
->setAttrib('placeholder', 'File containing template headers')
->setLabel('Template File <span class="glyphicons glyphicons-sheriffs_star requiredFlag"></span>')
->setDestination($destination)
->setRequired(true)
->addValidator('Count', false, 1)
->addValidator('NotEmpty', false)
->addValidator('Size', false, '15MB')
->addValidator("Extension", false, array('txt' ,'csv'))
->addFilter('StringTrim');