I can't say for sure this is correct, but your query doesn't look quite right. Specifically the way you are joining Answers to Users. To me it looks like there is not foreign key relation between them in that join, even though there may be in your schema.
Try this, it may be incorrect for your schema, but you should be able to understand what is going on. Assuming the join is the issue, the ordering should work.
use Doctrine\Common\Collections\Criteria;
$qb
->select('u')
->from('EventManager_Entity_User', 'u')
->join('u.answers', 'a')
->join('a.question', 'q')
->where('q.id = :question_id')
->setParameter('question_id', $qid)
->orderBy('a.answer', Criteria::ASC)
;
I changed it to an inner join, without it some users not linked to answers not may be ordered like the rest.