If you are going to route from (:any) => site
you have to specify a higher priority route for the admin controller that overrides that route for those pages.
$route['default_controller'] = "site";
$route['admin/(:any)'] = "admin/$1";
$route['(:any)'] = "site/$1";