If you have debug=true in app/config/app.php (or a local environment's config) you will not be cacheing anything. If you aren't using the cache then it makes sense that your load times with apc vs file cacheing would be similar.
First, try setting debug to false globally, or for your local environment. Then run php artisan optimize after cacheing is disabled.
Test your speeds using a direct route (no controller, simply return "some string"; from the routes.php route for the homepage.
Try returning the same string from a controller action. Map this action to the same route for the homepage and compare. On my local setup I see about a 10ms difference.
If that doesn't speed your app up then try installing the profiler suggested (or the one I prefer: https://packagist.org/packages/sebklaus/profiler) and see what is taking the most time to run. You can enable either profiler so that they run even when debug mode is false.
Another alternative is not to worry about local speed or speed during development and get laravel sped up once your app is working as you want.
Here are some tips to help you with the post-development optimization: Optimizing for production with Laravel 4