Why is creating a large number of child sites becoming slower?

Question

I need to create about 8000 sites in a site collection, based on Title and path attributes from a database table with 8000 records. I wrote a simple console based application in C# that utilizes the Server Object Model and the following algorithm:

1. Connect to this database and retrieve all the rows in this table
2. Connect to the desired SharePoint Site Collection (SPSite)
3. For each row in rows, Add a website under SPSite.RootWeb.Webs using Add method

This application is running on one of the web servers, not sure if that matters. Actual code for child site creation:

/*----
 * Create the child web in the site collection.
 *----*/
using (SPSite siteCollection = new SPSite(siteCollectionUrl))
{
    using (SPWeb rootWeb = siteCollection.RootWeb)
    {
        // Create a new child web in the site collection using the input parameters defining the child web.
        SPWeb newWeb = rootWeb.Webs.Add(webUrl, title, description, 1033, webTemplate, useUniquePermissions, false);
        newWeb.Dispose();
     }
 }

The setup is a medium SharePoint 2016 farm with two app servers (MinRole: App + Search) and two web servers (MinRole: Web + Distributed Cache).

The application is still running. At first, creation of the child site was taking about 15 seconds. After about 500 sites, that same operation is taking 40 seconds. Now, at about 1500 sites, site creation is taking 70 seconds. If this pattern keeps growing, I am afraid it will take way too long to create all 8000 sites. What may be the reason for this slow down? Is there any way to optimize and increase the speed of site creation, hopefully dramatically? Because I have to provision several more site collections that are similarly large.

Update 1/20/2018: I re-attempted the same site collection provisioning with a PowerShell based script:

Add-PSSnapin Microsoft.SharePoint.PowerShell -ErrorAction Stop
$ErrorActionPreference = "Stop"

$LogFile = "C:\Development\TestLargeSiteCollectionCreation_$(Get-Date -Format "yyyyMMdd").log"

Function WriteToLog
{
  #[CmdletBinding()]
  Param([string] $logdata)

  Write-Host $logdata; Out-File -filepath $LogFile -inputobject $logdata -Append
}

# Create site collection
$SiteCollUrl = "http://<portal-url>/alsc"
$SiteCollTitle = "A Large Site Collection"
$SiteOwner = "CONTOSO\spadmin"
$WebTemplate = Get-SPWebTemplate "STS#0"

$SiteColl = Get-SPSite -Identity $SiteCollUrl -ErrorAction SilentlyContinue
if (!$SiteColl) {
    $result = New-SPSite -Name $SiteCollTitle -Url $SiteCollUrl -Template $WebTemplate -OwnerAlias $SiteOwner
    WriteToLog "Created site => $result"
}
else {
    WriteToLog "Found site => $SiteColl"
}

Get-Content "C:\Development\childSiteList.txt" | ForEach-Object {
    $childpath = $_

    $c = Start-SPAssignment
    $mc = Measure-Command { $newChildWeb = New-SPWeb -Name $childpath -Url "$SiteCollUrl/$childpath" -Template $WebTemplate -UniquePermissions -UseParentTopNav -AssignmentCollection $c -ErrorAction SilentlyContinue }
    if ($newChildWeb) {
        WriteToLog "Created case site $SiteCollUrl/$childpath in $($mc.Seconds) second(s)"
    }
    else {
        WriteToLog "Unable to create case site $SiteCollUrl/$childpath"
    }
    Stop-SPAssignment $c
}

I was previously told by a MS SharePoint Support engineer that SharePoint PowerShell cmdlets are preferred to SharePoint Object Model for this type of provisioning, which prompted me to try the above approach. The script ran marginally faster, but I observed that the rate at the beginning was 11 seconds per child site and by the time the script ended, the rate averaged at 40 seconds.

Does a moderately sized farm comprising of 2 servers with MinRole of Web Front End plus Distributed Cache and 2 other servers with MinRole of Application plus Search and finally a 2-server SQL Server cluster somehow have anything to do with this dismal performance? I have a staging environment with a similar set up but comprising of one server each, that is significantly faster and the rate is uniformly set at 10-11 seconds per site.

Second Update 1/21/2018: Corresponding with the creation of each subsite, I see the following error in Event Viewer (TestLargeSiteCollectionCreation is the name of the C# console application I wrote to provision a large number of sub-sites in an existing site collection):

A failure was reported when trying to invoke a service application:

EndpointFailure

Process Name: TestLargeSiteCollectionCreation

Process ID: 6428

AppDomain Name: TestLargeSiteCollectionCreation.exe

AppDomain ID: 1

Service Application Uri: urn:schemas-microsoft-com:sharepoint:service:d623596b16ff4873bac023833775c635#authority=urn:uuid:92c677282f3f411185886510365a07db&authority=https:TestLargeSiteCollectionCreationFTestLargeSiteCollectionCreationF:32844TestLargeSiteCollectionCreationFTopologyTestLargeSiteCollectionCreationFtopology.svc

Active Endpoints: 3

Failed Endpoints:1

Affected Endpoint: http://:32843/d623596b16ff4873bac023833775c635/ProfileService.svc

As I mentioned previously, the production environment has 4 MinRole servers - App01/02 are App+Search, Web01/02 are Web+Distributed-Cache.

Answer 1

I can't tell from your code snippets, but are you creating the SPSite and the SPWeb in each iteration? If so, those are both "non managed memory" objects and will create a lot of objects to be cleaned up by the GC in the background.

Are you using this:

foreach (row in rows) 
{
  using (SPSite siteCollection = new SPSite(siteCollectionUrl))
  {
    using (SPWeb rootWeb = siteCollection.RootWeb)
    {
        // Create a new child web in the site collection using the input parameters defining the child web.
        SPWeb newWeb = rootWeb.Webs.Add(webUrl, title, description, 1033, webTemplate, useUniquePermissions, false);
        newWeb.Dispose();
     }
  } 
}

or this:

using (SPSite siteCollection = new SPSite(siteCollectionUrl))
{
  using (SPWeb rootWeb = siteCollection.RootWeb)
  {
    foreach (row in rows) 
    {
        // Create a new child web in the site collection using the input parameters defining the child web.
        SPWeb newWeb = rootWeb.Webs.Add(webUrl, title, description, 1033, webTemplate, useUniquePermissions, false);
        newWeb.Dispose();
     }
  } 
}

My tests were on a VM-hosted single server farm. The VM is running on a laptop, so not too much performance here. Each subsite took an average of 18 seconds to create, and while a bit random (17.5 - 18.25 seconds), showed no up or down trend. There were occasional higher times (25 seconds for example), which is to be expected on a server running background processes and timer jobs. RAM for the Console app grew from 300MB to around 1 GB during the run. (I paused it once, and RAM started going down, so the GC is not keeping up.) I only ran this for 500 loops as you saw a big difference by then.

UPDATE: I was running the console application in debug mode. Without debug mode the time to create a subsite is just over 7 seconds, and is relatively consistent. (No upward trend.)

Here's my code:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.SharePoint;

namespace ConsoleApplication7
{
    class Program
    {
        static void Main(string[] args)
        {
            string siteCollectionUrl = "http://sp2016/sites/manysubsites";
            using (SPSite siteCollection = new SPSite(siteCollectionUrl))
            {
                using (SPWeb rootWeb = siteCollection.RootWeb)
                {
                    string webTemplate = rootWeb.WebTemplate;
                    bool useUniquePermissions = false;

                    for (int i = 0; i < 500; i++)
                    {
                        DateTime t1 = DateTime.Now;
                        string title = "sub" + i.ToString().PadLeft(4).Replace(" ", "0");
                        string description = title;
                        string webUrl = title; 

                        // Create a new child web in the site collection using the input parameters defining the child web.
                        SPWeb newWeb = rootWeb.Webs.Add(webUrl, title, description, 1033, webTemplate, useUniquePermissions, false);
                        newWeb.Dispose();
                        DateTime t2 = DateTime.Now;
                        Console.WriteLine("Site: {0}, Time: {1}", title, (t2 - t1));
                    }
                }
            }
        }
    }
}

Other things to consider:

• Templates and features. Which template are you using? Does it have Features that trigger additional work in the creation of the web?
• Search. Each site will be crawled by search. Can you pause search while creating these sites and see if that makes a difference? (Search is paused on my test VM.)
• Custom Tasks or Event Receivers that are responding to new SPWebs.
• Other activity on the production servers. What % CPU and RAM are being used, both before and during running of your code? (During my tests I saw little change in CPU and RAM over the run of the code, which implies the biggest impact is on the SQL server, not the web servers.)
• The 5000 limit. Once you reach the 5000 limit, pages like Site Contents may no longer be able to display the list of the subsites. (I'll rerun my code to create more than 5000 to see what else is impacted.)
• You are using external blog storage... I do not have that in my tests.

So, my "answer" is:

• If you are not recreating the SPSite or SPWeb (for the rootweb) in each loop, then it's not your code or the basic process.
• It could be search.
• It could have something to do with the load balanced servers. (creating locks on each other? Unlikely, but possible.)
• It could have to do with a Feature in the template.

Answer 2

Can you check the usage report on your site collection and if you can disable that during the site creation time. Increase the site Quota until the script finish might help as well. You can stop the search service and search crawl during the script running if possible, for sure you must stop the incremental crawl if it is running. You can increase the speed 10 times more if you go with multithreading, call the site creation method in multithreading, as shown below. But first you can start with two, four, and six and gradually increase the number of thread, be aware about your server memory and resource as it will consume much resource. Please check if haven’t turned the Second Stage Recycle bin off or set a site quota. If a site quota has not been set then the growth of the Second Stage Recycle bin is not limited... The second stage recycle bin is by default limited to 50% size of the site quota, in other words if you have a site quota of 100 GB then you would have a Second Stage recycle bin of 50 GB. If a site quota has not been set, there are not any growth limitations...

Param($Command = $(Read-Host "Enter the script file"), [Parameter(ValueFromPipeline=$true,ValueFromPipelineByPropertyName=$true)]$ObjectList, $InputParam = $Null, $MaxThreads = 10, $SleepTimer = 200, $MaxResultTime = 120, [HashTable]$AddParam = @{}, [Array]$AddSwitch = @() ) https://gallery.technet.microsoft.com/scriptcenter/Multi-threading-Powershell-d2c0e2e5