Your own build server does not make use of the Hosted Build Cluster of VSO and thus does not count towards your build credits. I can't find any clear indication on the VSO site that an on-premise server is free, only that you're charged for the "Shared Resources" used by your VSO project.
Shared Resources (per Visual Studio Online account) - A Visual Studio Online account includes resources that are shared across all users and projects on that account.
Build - Build definitions can be created using Visual Studio and can be run on Visual Studio Online. Builds can be configured to run on demand, at a specified interval, or for continuous integration (CI). Build is charged per minute for the actual amount of computing time it takes to build the project.
http://www.windowsazure.com/en-us/pricing/details/visual-studio-online/
Your on premise build "does not run on VSO". And this is the closest wording to a confirmation or denial. But I can tell from experience that the on premise server indeed does not count towards your "Hosted Build" minutes.