In your composer.json you have by default a lot of libs, you simply have to remove what you won't need. For example,
- symfony/monolog-bundle is for logging
- symfony/swiftmailer-bundle is for sending emails
- doctrine/orm and doctrine/doctrine-bundle are for handling objects persistence
- jms/security-extra-bundle enhances the security component
Anything could be customized basing on your needs by simply removing the key:value pair in this file and later
$ cd your_project_dir; php app/console composer.phar update
It's not correct to simply remove the directories under vendor/, because composer reads from composer.json and re-creates the vendor/* dir every time you run it.
Linuxatico
Edit: one alternative is to use Silex, it's a php microframework based on Symfony2