The CreateBuildDetailsSpec method has an overloaded method that takes in TeamProject as a parameter. See this
To search for a particulra date, set the MinFinishTime and MaxFinishTime properties of buildDetailsSpec object. Something like below (assuming that dateToGet has the 00:00:00 hours)
var buildDetailSpec = buildServer.CreateBuildDetailSpec(project.Name);
buildDetailSpec.InformationTypes = null;
buildDetailSpec.MaxFinishTime = dateToGet; /*at 00:00:00*/
buildDetailSpec.MaxFinishTime = dateToGet.AddDays(1).AddMinutes(-1); /*at 23:59:59*/
IBuildDetail[] builds = buildServer.QueryBuilds(buildDetailSpec).Builds;