Scanning 10 million folders and only zipping the childless folders

Question

My scenario is that I have 300 million email messages, sorted throughout 10 million folders on an external QNAP NAS. Files are only stored in the lowermost, childless folder, usually 6 folders deep. My problem now is that I need to move each childless subfolder out of the NAS, compress it, and then move it back in as a self named zip.

Trying to do this with powershell, but unfortunately the server is an old dog that needs to be taken down old yeller style, and is running win 2k3. If it can be done easier or more efficiently on the QNAP's Linux system, that would be cool. But to be honest, installing Samba on that to connect to a windows share would be a challenge for me. The NAS currently has 0 bytes free. :)

I wrote up this powershell script, but it's still queuing up child-item folders and is at 3GB of RAM so far. I'm anticipating that it will hit a limit and fail.

#Script to clean up the OrchestriaCache NAS

$pwdZ = 'c:\temp\test\'
$zip = 'c:\temp\7z.exe'
$dest = 'c:\temp\zip'

$a = get-childitem $pwdZ -recurse | where-object {$_.PSISContainer -eq $true}
$b = $a | where-object {$_.GetFiles().Count -ge 1} #| select-object FullName

write-host $b.FullName
$ctr = 0
$cnt = $b.Count
$b | foreach-object { 
    $ctr++
    write-host $('[' + $ctr + '\' + $cnt + '] Zipping: ' + $_.fullname)     -foregroundcolor red
move-item -path $($_.Fullname + '\*.*') -force -destination $dest
cmd /c $('"c:\temp\7z.exe a ' + $($_.FullName + '\' + $_.Name + '.zip') + ' '+ $($dest + '\*"'))
}

How can this be improved? I was thinking of starting one external powershell process for all 400+ root subfolders, but that would create too much IO contention against the NAS.

Edit: Got the out of memory error I was dreading:

Where-Object : Exception of type 'System.OutOfMemoryException' was thrown.
At D:\BKeys\CacheCleanup.ps1:7 char:48
+ $a = get-childitem $src -recurse | where-object <<<<  {$_.PSISContainer -eq $
true}
    + CategoryInfo          : NotSpecified: (:) [Where-Object], OutOfMemoryExc
   eption
    + FullyQualifiedErrorId : System.OutOfMemoryException,Microsoft.PowerShell
   .Commands.WhereObjectCommand

You cannot call a method on a null-valued expression.
At D:\BKeys\CacheCleanup.ps1:8 char:36
+ $b = $a | where-object {$_.GetFiles <<<< ().Count -ge 1} #| select-object Ful
lName
    + CategoryInfo          : InvalidOperation: (GetFiles:String) [], RuntimeE
   xception
    + FullyQualifiedErrorId : InvokeMethodOnNull


[1\] Zipping:

Answer 1

Not tested, but this should be easier on your memory load:

#Script to clean up the OrchestriaCache NAS

$pwdZ = 'c:\temp\test\'
$zip = 'c:\temp\7z.exe'
$dest = 'c:\temp\zip'

#$a = get-childitem $pwdZ -recurse | where-object {$_.PSISContainer -eq $true}

#$b = $a | where-object {$_.GetFiles().Count -ge 1} #| select-object FullName

$b = cmd /c dir $pwdZ /b /s /ad |
      where-object {([IO.Directory]::GetFiles($_)).Count -ge 1} 


#write-host $b.FullName
write-host $b

$ctr = 0
$cnt = $b.Count

$b | foreach-object {
    $ctr++
    write-host $('[' + $ctr + '\' + $cnt + '] Zipping: ' + $_)     -foregroundcolor red
    $Name = $_.split('\')[-1]

move-item -path $($_ + '\*.*') -force -destination $dest
cmd /c $('"c:\temp\7z.exe a ' + $($_ + '\' + $Name + '.zip') + ' '+ $($dest + '\*"'))
}

The legacy dir command is much faster and less memory intensive than get-childitem for large directory structures. The /b /s /ad switches will make it recurse and return only the fullname strings of the directories.

The intermediate variable $a holding the complete list of directories is eliminated by filtering the folders that don't contain files as they come in. The name is parsed from that by splitting on the backslashes and taking the last element. You could refine this a little further by making your zip routine a filter or pipeline function so that you start zipping files as soon as you find a directory that has files in it. If you add additional code to save a list of directories you've already zipped, or check for the presence of a .zip file already in the directory it can be made to be re-startable.

Answer 2

You asked for a powershell script but I guess you just want to get the job done.

A batch script is probably going to do this fairly easily and with low specs, if you prepare and make enough room for the first few zips.

Add your 7-zip command to the inner loop, and del the *.eml or whatever files.

@echo off
pushd "\\server\share" || goto :EOF
   for /d /r %%a in (*) do (
     echo processing "%%a"
       pushd "%%a"
         dir /b /ad 2>nul | findstr "^" >nul || (
             echo Folder "%%a" has no subdirectories
             echo 7zip with *.eml 
             echo and then nuke the files
             )
       popd
  )
popd
echo done
pause