I'd recommend using the event/observer system in Magento. Inchoo has a post on how to use this to limit orders over a certain cost.
http://inchoo.net/ecommerce/magento/magento-maximum-allowed-order-amount/
You could do something very similar but check for the number of items in the basket instead.