SharePoint sitemap0.xml Missing Content

Question

I am having issues with certain sub-sites on a SharePoint site not showing up in the sitemap0.xml/sitemap_mobile0.xml files that are generated by SharePoint. Some sub-sites show up and some don't for some reason in the sitemap files.

I have a site collection and many sub-sites under the main site. Anonymous Access is enabled for both the Web Application (through Central Admin) and Site Collection are enabled. Search has been configured for the environment and all the pages/sub-site contents show up when searching for them through the Search Box/Search Results web parts. At the Site Collection Level, I have enabled the feature Search Engine Sitemap. I have also run both incremental and full crawls for the SharePoint site (no major errors). The Edit SEO Properties have also been double checked on the pages that don't show up in the SharePoint generated sitemap file and I've made sure that the Include in sitemap and expose to Search Engines option is selected. I've deleted all the sitemaps and regenerated them re-running the Search Engine Sitemap job in Central Admin for the following files multiple times:

sitemap.xml, sitemap_tmp.xml, sitemap0.xml, sitemap0_tmp.xml, sitemap_mobile0.xml, sitemap_mobile0_tmp.xml

And lastly, I've double-checked the robots.txt settings through Site Settings->Search Engine Sitemap Settings and for testing, all I have in there is:

User-agent: *

*So there are no disallows currently.*

I guess I'm not sure what else to check to ensure that all sub-sites on a SharePoint site is included in the SharePoint generated sitemap files. All sub-sites do have Anonymous Access enabled.

Any help would be appreciated and thanks for reading if you got through all of this.

Answer 1

Not sure what is happening specifically, but I've found I guess what could be considered an answer to this particular issue.

What I ended up doing is throwing every affected page in the Pages library for the affected sub-site to the recycle bin, then in the Central Admin I run an incremental crawl on the site's content source, then run the Search Engine Sitemap job for the site.

After that, restore the pages from the recycle bin, then repeat the steps above. All the pages that weren't showing up in the sitemap0.xml should now show up.

Not sure why this needs to be done, but so far this has worked on all the affected sub-sites I've tried this on. If anyone has a better solution or could elaborate on what is going on, it would be appreciated. Otherwise, this will work for my purposes.

Update 06-29-2017: Very possible this was the issue in the first place, but there is also a 500 entry limit on SharePoint 2013 by default on sitemaps. Then reading this other forum post, I believe the sitemap works exactly as expected. Then I ran this to update the limit:

$ssa = Get-SPEnterpriseSearchServiceApplication
$ssa.MaxRowLimit = 5000
$ssa.Update()

Possibly what was happening in my original answer was that when I was putting the pages in the recycle bin, then restoring them, the sitemap would recognize the "new" page(s) and put it in the sitemap. But at the same time, it would remove some other random entry/entries in the sitemap while getting these "new" page or pages for the sitemap as 500 entries is still the limit. Sounds strange, but seems like Microsoft isn't accommodating for larger sites.