Bulk Upload 1M files with SharePoint PnP PowerShell

Question

For the test purposes, we are trying to upload 1 million documents to document library in SharePoint Online. If everything goes really well, it will take approximately 7 days using our current script (see below).

Question: is there a bulk-upload technique we can use to generate a very large number of documents?

Current script that does not support bulk upload

# =================================================================== #
#              Script for Generating 1M Documents
# =================================================================== #
$SiteURL = "https://contoso.sharepoint.com/sites/ClassicTeam"
Connect-PnPOnline -Url $SiteURL -UseWebLogin

# =================================================================== #
#              Choose destination library
# =================================================================== #
$DocumentLibrary = 'Shared Documents'

# =================================================================== #
#               Upload 100K Documents
# =================================================================== #
$documentsToGenerate = 1000000

$files = Get-ChildItem -LiteralPath .\SampleFiles

$stopWatch = [system.diagnostics.stopwatch]::StartNew()
for($i=0;$i -lt $documentsToGenerate; $i++){
    $fileIndex = $i % $files.Count
    $localFile = $files[$fileIndex]
    $fineSuffix = (Get-Date).ToString("yyyyMMdd_hhmmss_ffff")
    $destinationFileName = $localFile.Name.Split(".")[0] +$fineSuffix + "." + $localFile.Name.Split(".")[1]
    $file = Add-PnPFile -Path $localFile.FullName -Folder $DocumentLibrary -NewFileName $destinationFileName
    Write-host [$i/$documentsToGenerate] " " -NoNewline -ForegroundColor Yellow
    Write-host  $file.Name "uploaded" -ForegroundColor Green 
}
$stopWatch.Stop()
Write-Host Time it took to create $documentsToGenerate documents in the $($SiteURL+$DestinationFolderUrl) -ForegroundColor Cyan
$stopWatch

Answer 1

Try this:

• create a sample document; for instance demodoc.pdf in a folder called DemoFolder
• use the below PowerShell script to multiply the document demodoc.pdf into one million copies, each with unique names:

$NumArray = (1..1000001)
ForEach ($number in $numArray) {
  Copy-Item "C:\Users\admin\Documents\DemoFolder\Demodoc.pdf" `
  -Destination "C:\Users\admin\Documents\DemoFolder\Demodoc$number.pdf"
}

• Then use the free SharePoint migration tool to migrate the one Million documents from the local computer to a document library in SharePoint online.

Obviously, the speed of the migration will depend on a number of factors such as the size of the documents, your internet connection, the speed of the local computer, etc. I tried this in a virtual machine on Microsoft Azure and the entire process was completed in 22 hours.