search results not including pages within folders

Question

The site I'm working on uses folders inside the Pages library (yeah, I know and agree.. but they pay the bills). When searching for key terms, I'm not getting the pages within these folders in my results.

Initially I think I was, but I added some crawl rules to hide list items, list forms, etc and having reset my index and done a full crawl, I only get pages that live in the Pages library itself, rather than within folders in that Pages library.

I removed my crawl rules, reset the index again (thankfully this is a dev site, not production) and I still only get pages, not folder/pages.

This is true with osssearchresults.aspx as well as my custom results.aspx file (which is essentially the ootb page with a different master)

Any ideas of what I can do to allow folder/page style results? I swear this was working before I add/removed the rules.

Kevin

[Updated to provide more info]

My site structure matches this example

www.mysite.com/...

.../Pages/default.aspx
.../us/Pages/default.aspx
.../us/Pages/products/default.aspx
.../us/Pages/products/allproducts.aspx
.../us/Pages/products/product1.aspx

Each of the pages within the us Pages library mentioned product1 in some way. So, when I search for product1 I would expect to see results for all 4 of the pages within that library, listed above.

What I see is just

.../us

no /us/Pages/default.aspx or any pages within the products folder inside that library.

strange, right?

Answer 1

Have you published and approved the content within the folders (and the folders themselves)? The crawler shouldn't index non-approved content.

Answer 2

I had the same problem here. We have an Enterprise Wiki with folders in the Pages library. We also have versioning and content approval activated on this library.

The crawler ignored files (Wikipages) in folders which haven't been published/approved yet.

After manually setting the Approval status to "Approved" on the folders the crawler found every page in it.

Hope that helps.