Using Visual Studio 2013, when I install a NuGet package, such as Microsoft.AspNet.Razor, into an ASP.NET web pages application then the bin folder is created.
Getting the binaries into the bin folder is part of the build process and it is not done by NuGet. NuGet can be used to restore the files into the packages folder but will not do anything with your bin folder when restoring.
For building and package restore to work it looks like you need to keep the bin folder and any .refresh files. You can remove the other binaries from your version control system.
System.Web.Razor.dll.refresh
The contents of this file tells Visual Studio where to find the NuGet packages:
..\..\Projects\MyWebSite\packages\Microsoft.AspNet.Razor.3.1.2\lib\net45\System.Web.Razor.dll
To test this, I deleted all the binaries from the bin folder, removed the packages folder, and then rebuilt the project. Visual Studio restores the packages and when building copies the required assemblies to the project's bin folder.