Matlab Distributed Server parfor can't find mex opencv files

Question

everyone!

I'm trying to parallelize an algorithm that uses mex files from mexopencv (KNearest.m, KNearest_.mexw32).

The program is based vlfeat (vlsift.mex32) + mexopencv (KNearest.m and KNearest_.mexw32).I classify descriptors obtained from images.

All the code is located on the fileshare

\\ LAB-07 \ untitled \ DISTRIB \ (this is the program code) 
\\ LAB-07 \ untitled \ + cv (mexopencv)

When I run the program with matlabpool close everything works well.

Then I make matlabpool open (2 computers on 2 cores each. ultimately 4 worker, but now I use for testing only 2 workers on the computer and run the program which) PathDependencises from fileshare -> \LAB-07\untitled\DISTRIB\ , \LAB-07\untitled+cv

Before parfor loop I train classifier on the local machine

classifiers = cv.KNearest
classifiers.train(Descriptors',Labels','MaxK',1)

Then run parfor

descr=vlsift(img);
PredictClasses = classifiers.predict(descr');

Error

Error in ==> KNearest>KNearest.find_nearest at 173
Invalid MEX-file '\\LAB-07\untitled\+cv\private\KNearest_.mexw32': 
The specified module could not be found.

That is KNearest.m finds, but no KNearest_.mexw32. Because KNearest_.mexw32 located in private folder, I changed the code KNearest.m (everywhere where it appeal KNearest_ () changed to cv.KNearest_ (). Example: this.id = сv.KNearest_ ()) and placed in a folder with KNearest_.mexw32 KNearest.m. As a result, get the same error

Immediately after matlabpool open file search on workers

pctRunOnAll which ('KNearest.m')
'KNearest.m' not found.
'KNearest.m' not found.
'KNearest.m' not found.

pctRunOnAll which ('KNearest_.mexw32')
'KNearest_.mexw32' not found.
'KNearest_.mexw32' not found.
'KNearest_.mexw32' not found.

after cd \LAB-07\untitled+cv

 pctRunOnAll which ('KNearest.m')
\\LAB-07\untitled\+cv\KNearest.m
\\LAB-07\untitled\+cv\KNearest.m  % cv.KNearest constructor
\\LAB-07\untitled\+cv\KNearest.m

>> pctRunOnAll which ('KNearest_.mexw32')
\\LAB-07\untitled\+cv\KNearest_.mexw32
\\LAB-07\untitled\+cv\KNearest_.mexw32
\\LAB-07\untitled\+cv\KNearest_.mexw32

I ran and FileDependecies, but the same result.

I do not know this is related or not, I display during the execution of the program classifiers after training and before parfor

classifiers = 

  cv.KNearest handle
  Package: cv

  Properties:
              id: 5
            MaxK: 1
        VarCount: 128
     SampleCount: 9162
    IsRegression: 0

  Methods, Events, Superclasses

Within parfor before classifiers.predict

classifiers = 

  cv.KNearest handle
  Package: cv

  Properties:
    id: 5

---

I tested the file cvtColor.mexw32. I left in a folder only 2 files cvtColor.mexw32 and vl_sift

parfor i=1:2
 im1=imread('Copy_of_start40.png');
 im_vl = im2single(rgb2gray(im1));
 desc=vl_sift(im_vl);
 im1 = cvtColor(im1,'RGB2GRAY');
end

The same error, and vl_sift work, cvtColor no...

Answer 1

If the worker machines can see the code in your shared filesystem, you should not need FileDependencies or PathDependencies at all. It looks like you're using Windows. It seems to me that the most likely problem is file permissions. MDCS workers running under a jobmanager on Windows by default run not using your own account (they run using the "LocalSystem" account I think), and so may well simply not have access to files on a shared filesystem. You could try making sure your code is world-readable.

Otherwise, you can add the files to the pool by using something like

matlabpool('addfiledependencies', {'\\LAB-07\untitled\+cv'})

Note that MATLAB interprets directories with a + in as defining "packages", not sure if this is intentional in your case.

EDIT

Ah, re-reading your original post, and your comments below - I suspect the problem is that the workers cannot see the libraries on which your MEX file depends. (That's what the "Invalid MEX-file" message is indicating). You could use http://www.dependencywalker.com/ to work out what are the dependencies of your MEX file, and make sure they're available on the workers (I think they need to be on %PATH%, or in the current directory).

Answer 2

Edric thanks. There was a problem in the PATH for parfor. With http://www.dependencywalker.com/ looked missing files and put them in a folder +cv. Only this method works in parfor.

But predict in parfor gives an error

PredictClasses = classifiers.predict(descr');

??? Error using ==> parallel_function at 598
Error in ==> KNearest>KNearest.find_nearest at 173
Unexpected Standard exception from MEX file.

What() is:..\..\..\src\opencv\modules\ml\src\knearest.cpp:365: error: (-2) The search
tree must be constructed first using train method

I solved this problem by calling each time within parfor train

classifiers = cv.KNearest
classifiers.train(Descriptors',Labels','MaxK',1)

But it's an ugly solution :)