First let me thank @Florian for his link and the nudge to get me digging a little :).
When following the instructions on sylius' getting started page:
$ wget http://getcomposer.org/composer.phar
$ composer.phar create-project sylius/sylius -s dev
$ cd sylius
$ app/console sylius:install
you are by default installing using the config_dev.yml file. This will setup your db as {dbname}_dev, as well as have debug on in app_dev.php. When debug is on "cache files are dynamically rebuilt on each request." This is what is causing the slow speed.
In regards to using app.php(production env) you can run app/console sylius:install -e prod. This will setup a regular db and you can use app.php.
Mystery solved :).
Useful resource: How to Master and Create new Environments - Symfony2