As Steven indicated in his answer the main reason for keeping the packages out of source control is to reduce the amount of data that needs to be stored by and transferred from/to the source control server. Obviously in a company where you control all the hardware neither the disk space nor the network transfer should be an issue, but why waste the disk space / time dealing with the packages if you don't need to. Note that the size of the package directory can be a quite considerable percentage of the total size of a workspace. In my case the packages directory takes up between 80% - 95% of the total workspace when using TFS (for my work workspaces) and between 50% - 75% for my private workspaces (which are based on git and thus have the .git directory which takes up some space). All in all a significant amount of space could be saved on your source control server if you use package restore.
One way to solve the access problem with Nuget.org is to have your own local package repository. This local package repository could be a local nuget web service or just a shared directory on a server. Because the repository lives inside your company LAN it should not be a big problem for the build server to get to the local nuget repository. A side benefit of this approach is that your build process is independent from Nuget.org (in the very rare case it goes down) and, more importantly, that you know exactly which packages get pulled into the build (because they will be the approved packages in your local repository).
For us the decision whether to use a local nuget repository and the package restore option depended on our decision to package all our internal libraries as nuget packages (I have described our development process in an answer to another nuget question). Because of that we needed a local package repository to distribute those internal packages. Which then meant that adding the third-party packages to this repository was easy and thus using package restore made perfect sense. If you don't package your internal libraries as nuget packages then you will put those in your source control along side your solution and code files. In that case you may as well do the same for the third-party libraries.
In the end it is all a trade-off. Disk space vs ease of infrastructure set-up etc. etc. Pick the solution that suits your environment best.