Give the full path instead, like
$filename = dirname(__FILE__) . '/upload/test.png';
Then try this,
if (is_file($filename)) {
chmod($filename, 0777);
if (unlink($filename)) {
echo 'File deleted';
} else {
echo 'Cannot remove that file';
}
} else {
echo 'File does not exist';
}