Have you tried with the following RewriteRule:
RewriteRule ^admin/?$ admin.php [L]
Your .htaccess would look likes:
# mod-rewrite engine
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} -s [OR]
RewriteCond %{REQUEST_FILENAME} -l [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^admin/?$ admin.php [L]
RewriteRule ^.*$ - [NC,L]
RewriteRule ^.*$ index.php [NC,L]
Update 1
and if you try the RewriteRules in the following order:
RewriteRule ^.*$ - [NC,L]
RewriteRule ^.*(admin).*$ admin.php [NC,L]
RewriteRule ^.*$ index.php [NC,L]