Custom Search webpart - How to 'drill down' into folders?

Question

We recently deployed a new custom Search Results webpart that utilizes a search scope we setup to query specific document libraries under a site hierarchy and return the results in a web part.

The web part results are showing that documents within folders are not being returned. Is there something that needs to be added to the search query in order to 'drill down' into the folders and add the documents within to the search results?

Relevant code below:

       private DataTable GetBindingData()
    {
        DataTable dt = new DataTable();

        try
        {
            SPSecurity.RunWithElevatedPrivileges(delegate()
              {
                  using (SPSite site = new SPSite(getSite()))
                  {
                      SPServiceContext serviceContext = SPServiceContext.GetContext(site);
                      SearchServiceApplicationProxy searchProxy = serviceContext.GetDefaultProxy(typeof(SearchServiceApplicationProxy)) as SearchServiceApplicationProxy;
                      if (searchProxy != null)
                      {
                          FullTextSqlQuery fullQuery = new FullTextSqlQuery(searchProxy);
                          fullQuery.QueryText = getFullTextQuery();
                          fullQuery.ResultTypes = ResultType.RelevantResults;
                          var queryResults = fullQuery.Execute();
                          var queryResultsTable = queryResults[ResultType.RelevantResults];
                          dt.Load(queryResultsTable, LoadOption.OverwriteChanges);
                          dt.Columns.Add("ModifiedDate");
                          dt = ModifyResults(dt);

                      }
                  }
              });
        }
        catch (Exception)
        {
        }

        return dt;
    }
    private string getFullTextQuery()
    {
        StringBuilder query = new StringBuilder();
        query.Append("SELECT Url, Title, Department, DepartmentService, Modified, DocumentType, FileLeafRef ");
        query.Append("FROM Scope() ");
        query.Append("WHERE ((\"SCOPE\" = 'OurDocuments')) ");

        query.Append(string.Format("AND CONTAINS (ContentType, 'OurDocs') "));

        string tax = ParentWebPart.taxValue;
        if (!string.IsNullOrEmpty(tax))
        {
            query.Append(string.Format("AND (PrimaryTopic = '{0}' ", tax));
            query.Append(string.Format("OR SecondaryTopic = '{0}')", tax));
        }

        return query.ToString();
    }

Answer 1

I figured this out. It seems there is a hard limit to the amount of items returned via the FullTextQuery (or the proxy - i cannot determine).

The 'fixed' code is below. A simple addition of an increase to the RowLimit (fullQuery.RowLimit = 1000;) was all I needed. Folders was a red herring:

        private DataTable GetBindingData()
    {
        DataTable dt = new DataTable();

        try
        {
            SPSecurity.RunWithElevatedPrivileges(delegate()
              {
                  using (SPSite site = new SPSite(getSite()))
                  {
                      SPServiceContext serviceContext = SPServiceContext.GetContext(site);
                      SearchServiceApplicationProxy searchProxy = serviceContext.GetDefaultProxy(typeof(SearchServiceApplicationProxy)) as SearchServiceApplicationProxy;
                      if (searchProxy != null)
                      {
                          FullTextSqlQuery fullQuery = new FullTextSqlQuery(searchProxy);
                          fullQuery.QueryText = getFullTextQuery();
                          fullQuery.ResultTypes = ResultType.RelevantResults;
                          fullQuery.RowLimit = 1000;
                          var queryResults = fullQuery.Execute();
                          noItemsLabel.Text = queryResults.Count.ToString();    //50!!??
                          var queryResultsTable = queryResults[ResultType.RelevantResults];
                          dt.Load(queryResultsTable, LoadOption.OverwriteChanges);
                          dt.Columns.Add("ModifiedDate");
                          dt = ModifyResults(dt);
                      }
                  }
              });
        }
        catch (Exception)
        {
        }

        return dt;
    }

Answer 2

To help troubleshoot the issue, I would suggest adding an out of the box results web part using that scope, and see if you get the same behavior or not. That will tell you if your scope definition is the issue, or the custom web part.

Typically the search engine shouldn't care about folders. I presume you specified web address paths in your search scope, typically they will include all subfolders, but I'd suggest reviewing them to see if there's some mistake there.

Also, are you sure the files are being excluded based on being in subfolders? Is it possible they are excluded for some other reason, for example I notice you are filtering further in your web part based on ContentType, PrimaryTopic and SecondaryTopic.