scrapyd is indeed a good way to go, max_proc or max_proc_per_cpu configuration can be used to restrict the number of parallel spdiers, you will then schedule spiders using scrapyd rest api like:
$ curl http://localhost:6800/schedule.json -d project=myproject -d spider=somespider