Directory.EnumerateFiles => UnauthorizedAccessException

Question

There is a nice new method in .NET 4.0 for getting files in a directory in a streaming way via enumeration.

The problem here is that if one wishes to enumerate all files one may not know in advance which files or folders are access protected and can throw an UnauthorizedAccessException.

To reproduce, one can just run this fragment:

foreach (var file in Directory.EnumerateFiles(@"c:\", "*", SearchOption.AllDirectories))
{
   // whatever
}

Before this .NET method existed it was possible to achieve roughly the same effect by implementing a recursive iterator on the string-array returning methods. But it's not quite as lazy as the new .NET method is.

So what to do? Can the UnauthorizedAccessException be suppressed or is a fact of life when using this method?

Seems to me that the method should have an overload accepting an action to deal with any exceptions.

Answer 1

Ths issue with the above answer is that is does not take care of exception in sub directories. This would be a better way to handling those exceptions so you get ALL files from ALL subdirectories except those with threw an access exception:

    /// <summary>
    /// A safe way to get all the files in a directory and sub directory without crashing on UnauthorizedException or PathTooLongException
    /// </summary>
    /// <param name="rootPath">Starting directory</param>
    /// <param name="patternMatch">Filename pattern match</param>
    /// <param name="searchOption">Search subdirectories or only top level directory for files</param>
    /// <returns>List of files</returns>
    public static IEnumerable<string> GetDirectoryFiles(string rootPath, string patternMatch, SearchOption searchOption)
    {
        var foundFiles = Enumerable.Empty<string>();

        if (searchOption == SearchOption.AllDirectories)
        {
            try
            {
                IEnumerable<string> subDirs = Directory.EnumerateDirectories(rootPath);
                foreach (string dir in subDirs)
                {
                    foundFiles = foundFiles.Concat(GetDirectoryFiles(dir, patternMatch, searchOption)); // Add files in subdirectories recursively to the list
                }
            }
            catch (UnauthorizedAccessException) { }
            catch (PathTooLongException) {}
        }

        try
        {
            foundFiles = foundFiles.Concat(Directory.EnumerateFiles(rootPath, patternMatch)); // Add files from the current directory
        }
        catch (UnauthorizedAccessException) { }

        return foundFiles;
    }

Answer 2

I Couldn't get the above to work, but here is my implementation, i've tested it on c:\users on a "Win7" box, because if has all these "nasty" dirs:

SafeWalk.EnumerateFiles(@"C:\users", "*.jpg", SearchOption.AllDirectories).Take(10)

Class:

public static class SafeWalk
{
    public static IEnumerable<string> EnumerateFiles(string path, string searchPattern, SearchOption searchOpt)
    {   
        try
        {
            var dirFiles = Enumerable.Empty<string>();
            if(searchOpt == SearchOption.AllDirectories)
            {
                dirFiles = Directory.EnumerateDirectories(path)
                                    .SelectMany(x => EnumerateFiles(x, searchPattern, searchOpt));
            }
            return dirFiles.Concat(Directory.EnumerateFiles(path, searchPattern));
        }
        catch(UnauthorizedAccessException ex)
        {
            return Enumerable.Empty<string>();
        }
    }
}

Answer 3

I understand it's MoveNext that throws the exception.

I tried to write a method that safe-walks a sequence and tries to ignore MoveNext exceptions. However I'm not sure if MoveNext advances position when it throws an exception, so this might as well be infinite loop. It is also bad idea because we would rely on implementation details.

But it's just so much fun!

public static IEnumerable<T> SafeWalk<T> (this IEnumerable<T> source)
{
    var enumerator = source.GetEnumerator();
    bool? hasCurrent = null;

    do {
        try {
            hasCurrent = enumerator.MoveNext();
        } catch {
            hasCurrent = null; // we're not sure
        }

        if (hasCurrent ?? false) // if not sure, do not return value
            yield return enumerator.Current;

    } while (hasCurrent ?? true); // if not sure, continue walking
}

foreach (var file in Directory.EnumerateFiles("c:\\", "*", SearchOption.AllDirectories)
                              .SafeWalk())
{
    // ...
}

This will only work if the following conditions are true about framework's implementation of this iterator (see FileSystemEnumerableIterator<TSource> in Reflector for reference):

• MoveNext advances its position when it fails;
• When MoveNext fails on last element, subsequent calls will return false instead of throwing an exception;
• This behavior is consistent for different versions of .NET Framework;
• I haven't made any logic or syntax mistakes.

Even if it works, please, never use it in production!
But I really wonder if it does.

Answer 4

I'm late, but I suggest using observable pattern instead:

public class FileUtil
{
  private static void FindFiles_(string path, string pattern,
    SearchOption option, IObserver<string> obs, CancellationToken token)
  {
    try
    {
      foreach (var file in Directory.EnumerateFiles(path, pattern,
        SearchOption.TopDirectoryOnly))
      {
        if (token.IsCancellationRequested) break;
        obs.OnNext(file);
      }

      if (option != SearchOption.AllDirectories) return;

      foreach (var dir in Directory.EnumerateDirectories(path, "*", 
        SearchOption.TopDirectoryOnly))
      {
        FindFiles_(dir, pattern, option, obs, token);
      }
    }
    catch (UnauthorizedAccessException) { }
    catch (PathTooLongException) { }
    catch (IOException) { }
    catch (Exception err) { obs.OnError(err); }
  }

  public static IObservable<string> GetFiles(string root, string pattern,
    SearchOption option)
  {
    return Observable.Create<string>(
      (obs, token) =>
        Task.Factory.StartNew(
          () =>
          {
            FindFiles_(root, pattern, option, obs, token);
            obs.OnCompleted();
          },
          token));
  }
}

Answer 5

I made my own Implementation of a Class that works arround this, as previous answers didnt seem to do what i wanted to. This simply skips all Files and Folders that it can not access, and returns all files that it could access.

public static class SafeWalk
{
    public static IEnumerable<string> EnumerateFiles(string path, string searchPattern, SearchOption searchOpt)
    {
        if (searchOpt == SearchOption.TopDirectoryOnly)
        {
            return Directory.EnumerateFiles(path, searchPattern, SearchOption.TopDirectoryOnly);
        }

        List<string> folders = new List<string>() { path };
        int folCount = 1;
        List<string> files = new List<string>() { };

        for (int i = 0; i < folCount; i++)
        {
            try
            {
                foreach (var newDir in Directory.EnumerateDirectories(folders[i], "*", SearchOption.TopDirectoryOnly))
                {
                    folders.Add(newDir);
                    folCount++;
                    try
                    {

                        foreach (var file in Directory.EnumerateFiles(newDir, searchPattern))
                        {
                            files.Add(file);
                        }
                    } catch (UnauthorizedAccessException)
                    {
                        // Failed to read a File, skipping it.
                    }
                }
            }
            catch (UnauthorizedAccessException)
            {
                // Failed to read a Folder, skipping it.
                continue;
            }
        }
        return files;
    }
}

Usable exactly like the regular EnumerateFiles function, simply using SafeWalk.EnumerateFiles(...) instead of Dictionary.EnumerateFiles(...)

Answer 6

Posting as an answer since I don't have the rep to add a comment let alone edit existing answers. My requirement was to minimize memory allocations, redundant variables, and have the system do a single enumeration of the directory.

static IEnumerable<string> FindFiles(string path, string filter = "*", bool recursive = false)
{
    IEnumerator<string> fEnum;
    try
    {
        fEnum = Directory.EnumerateFiles(path, filter, recursive ? SearchOption.AllDirectories : SearchOption.TopDirectoryOnly).GetEnumerator();
    }
    catch (UnauthorizedAccessException) { yield break; }
    while (true)
    {
        try { if (!fEnum.MoveNext()) break; }
        catch (UnauthorizedAccessException) { continue; }
        yield return fEnum.Current;
    }
}

Dan Bechard mentions in a comment:

Unfortunately, MoveNext() doesn't advance its position when it throws an exception.

This may have been fixed in either a newer release of .Net or in a Windows 10 build? I'm not having this issue in .NET 5.0 on Windows 10. Tested by searching my entire system drive.