Why does inserting list items into a sharepoint site get slower and slower over time?

Question

I am inserting some test data into my single server sharepoint farm. I noticed some performance decay after you insert a bunch of items.

Here is the powershell script:

Add-PSSnapin Microsoft.SharePoint.PowerShell 

# Set the variables
$WebURL = "http://win-d9fm7ip9r36/sites/adventureworks-cycles-inc/operati"
$DocLibName = "Shared Documents"

# Get a variable that points to the folder
$Web = Get-SPWeb $WebURL
$List = $Web.GetFolder($DocLibName)
$Files = $List.Files

For ($i=20001; $i -le 200000; $i++) {
    $FilePath = "c:\testdir\File$i.txt"
    echo "file $i" > $FilePath

    # Get just the name of the file from the whole path
    $FileName = $FilePath.Substring($FilePath.LastIndexOf("\")+1)

    # Load the file into a variable
    $File= Get-ChildItem $FilePath

    # Upload it to SharePoint
    $Files.Add($DocLibName +"/" + $FileName,$File.OpenRead(),$false)
}
$web.Dispose()

It runs very fast at first, then starts to slow down. After a couple hours of running, it becomes extremely slow. Only manages to add a 10,000 in a 8 hour period.

The whole box becomes pretty slow after running that script for a while. The virtual machine has 2 cpu's and they are both 100% while inserting list items.

I noticed that if i start inserting into a new list, it gets faster temporarily and then eventually slow again.

So it seems like something is slow about inserting list items into a list with lots of items.

Any idea how I can eliminate this CPU churn?

Answer 1

Instead of $Files.Add use the list.AddItem();

The big problem is this object $Files = $List.Files. It means you hold information about all files in the list in memory changing to list.AddItem() should really help

Have a look at this other thread for more insights: Best practice to add list item in SharePoint Server OM?

Another element that might be slowing you down is crawler. If you are adding new content and crawler starts to crawl same library it may significantly impact performance. In case of migration best practice is to pause crawler until you complete migration.