You can use AppHarbor's solution file convention to achieve this. If you have a separate project containing the specflow/selenium tests the trick is to make sure that this project is not referenced in the solution we build.
More specifically you can create a solution with the name AppHarbor.sln
and reference only the web, console and test projects you want AppHarbor to build and test from this solution file. AppHarbor will always prefer building a solution with that name if it's found in your repository.
You can then maintain a separate solution file (likely the one you're already using) for development where the project containing the specflow/selenium tests is included. This will allow you to execute your tests locally before committing and retain your current workflow.