It works, but you expect something else.
The output is interpreted by your browser as HTML. You can get the plain output by setting the content type to plain or using a command line.
<?php
header('Content-Type: text/plain');
var_dump(htmlspecialchars("<a href='test'>Test</a>", ENT_QUOTES, 'UTF-8'));
echo htmlspecialchars("<a href='test'>Test</a>", ENT_XHTML, 'UTF-8');