This can be done simply enough. Create a set (that can be much more efficiently searched than a List
or array) of all of the paths in the second directory. Then you can easily write a query to give you all of the items where the directory path is in that set (or not in that set, if that's what you want). Note the Comparer is passed to the set to ensure it does a case insensitive comparison.
var directoryBPaths = new HashSet<string>(dInfoB.Select(dir => dir.FullName),
StringComparer.InvariantCultureIgnoreCase);
var directoriesToRemove = dInfoA.Where(dir =>
directoryBPaths.Contains(dir.FullName));
//If you just want a sequence of the items not
//in the other set, get that directly
var directoriesToKeep = dInfoA.Where(dir =>
!directoryBPaths.Contains(dir.FullName));