Just use a socket instead of the IP address? See the manual for an example. This circumvents the network stack altogether.
Put something like
listen=/tmp/mysql_sphinx.sock
into the sphinx.cnf and put the same socket into the DNS of PDO like
$db = new PDO('mysql:dbname=testdb;unix_socket=/tmp/mysql_sphinx.sock');
This frees you from the need to make MySQL accessible over the network too. This is better taking security into account.