Homebrew actually has a formula for vtk5 separately now, so this should now be the correct way to install it (currently it's 5.10.1).
brew install vtk5 --with-qt
You also need eigen, flann, qhull
and boost
. Also useful, but optional is getting sphinx
via pip install sphinx
if you want docs. If you want to build simulations (see below option) then you need to get glew
too (although it wouldn't find the header).
I can verify that this works on 10.8 and will check 10.9 tonight. I've had problems installing 1.7.1 from the archive, but cloning the github repository worked for me.
$ git clone https://github.com/PointCloudLibrary/pcl pcl-trunk
$ cd pcl_trunk
$ mkdir build; cd build
$ cmake .. -DBUILD_global_tests=ON -DBUILD_examples=ON -DBUILD_simulations=ON
$ ./configure; make; make install
Without VTK you can get to a few % before the PCL install fails. Installing VTK with X11 seems to be a recipe for a disaster here, I had to manually install libX11, libXt and others because VTK was trying to include the incorrect headers. The problems continued with OpenGL and I couldn't compile past around 42%. I reinstalled vtk5 without --with-x11
and it compiled fully.
There is a PCL formula for Homebrew knocking around, but it's rarely updated and I didn't have much luck editing it (not familiar enough with Brew's syntax). I tried to update it and submit a pull request, but when it came to testing the actual 'stable' build fails at the moment so there's no point.
EDIT: To be clear, installing the 1.7.1 tarball fails due to some problem with Eigen (around 12% in when compiling pcl_search).
Proposed pull request, it's a HEAD only fix at the moment. https://github.com/ros/homebrew-hydro/pull/7