Using R's GPU packages on Amazon

Question

I spent almost a whole day trying to get this running and finally decided to come to SO because there are going to be people here who have tried this =)

I would like to get an Amazon-EC2 GPU machine running with rpud (or another R GPU package), either a cg1.4xlarge or g2.2xlarge as those are the only two GPU machines Amazon(AWS) has.

This post How to run a GPU instance using Amazon EC2 Panel? helped me realize that I couldn't just change my rstudio-server machine to a gpu machine as I was using an incorrect AMI.

Started out with this AMI from Amazon CentOS 6 (x86_64) - with Updates
So I decided I had to build my own and started following the directions here http://www.r-tutor.com/gpu-computing/rpud-installation. As well as http://www.louisaslett.com/Talks/GPU_Programming_Basics_Getting_Started/Handout.pdf (Louis Aslett is the same guy who has an amazing demo of using rstudio-server on an ec2 (http://www.louisaslett.com/RStudio_AMI/).

Both of those end up leading you to here: http://docs.nvidia.com/cuda/cuda-getting-started-guide-for-linux/index.html#package-manager-installation, so you can get the CUDA toolkit (which seems to be a necessity for GPU computing). Some of the AWS machines already have a Cuda toolkit installed, however I figured I would follow the directions on my first try (even though they are installed, sometimes the versioning of stuff hasn't been the same as some of the tutorials I've found). So I follow those directions to a T and I end up getting a few errors like nvidia-settings-319.37-30.fc18.x86_64 (cuda) Requires: libgdk-x11-2.0.so.0()(64bit) when I try to run the examples with $ cuda-install-samples-5.5.sh <dir>. So I download all of those required packages. I try again and end up with rpud errors (will detail those later).

This machine is Ubuntu-12.04
So of course I think, someone must have already built an AMI for this and find someone who did it with python. I might just end up using python anyway because of more speed bonuses, but at this point I want to get it working with R. Here: http://vasir.net/blog/opencl/installing-cuda-opencl-pyopencl-on-aws-ec2 is the link to the AMI/ instructions on how to build it from scratch if you want to. Of course, then you have to install R, which defaults to R2.14, so following this great post How to install R version 3 to get R.3.1.0 running because rpud requires R>=R.2.8.

Following the directions from the r-tutor site listed above (either from simply using install.packages('rpud') or the $ R CMD INSTALL rpud_<version>.tar.gz or by using the directions http://cran.r-project.org/web/packages/rpud/INSTALL, I get this message:

* installing *source* package ‘rpud’ ...
checking "environment variable CUDA_HOME"... "CUDA_HOME not set; using default /usr/local/cuda"
checking for /usr/local/cuda/bin/nvcc... yes
"nvcc found"
checking "whether this is the 64 bit linux version of CUDA"... checking for /usr/local/cuda/lib64/libcublas.so... yes
"yes -- using /usr/local/cuda/lib64 for CUDA libs"
"using -I/usr/share/R/include for R header files"
"using -Wl,--export-dynamic -fopenmp  -L/usr/lib/R/lib -lR -lpcre -llzma -lbz2 -lrt -ldl -lm for R shared libraries"
configure: creating ./config.status
config.status: creating src/Makefile
** libs
** arch -
/usr/local/cuda/bin/nvcc -c -I/usr/local/cuda/include -Xcompiler "-I/usr/share/R/include -fpic" rpud.cu -o rpud.o
/usr/local/cuda/bin/nvcc -c -I/usr/local/cuda/include -Xcompiler "-I/usr/share/R/include -fpic" rpudist.cu -o rpudist.o
rpudist.cu(159): warning: use of "=" where "==" may have been intended

rpudist.cu(159): warning: use of "=" where "==" may have been intended

ptxas /tmp/tmpxft_000006af_00000000-5_rpudist.ptx, line 904; warning : Double is not supported. Demoting to float
/usr/local/cuda/bin/nvcc -shared -Xlinker "-Wl,--export-dynamic -fopenmp  -L/usr/lib/R/lib -lR -lpcre -llzma -lbz2 -lrt -ldl -lm -Wl,-rpath,/usr/local/cuda/lib64" -L/usr/local/cuda/lib64 -lcublas -lcuda rpud.o rpudist.o -o rpud.so
/usr/bin/ld: unrecognized option '-Wl'
/usr/bin/ld: use the --help option for usage information
collect2: ld returned 1 exit status
make: *** [rpud.so] Error 1
ERROR: compilation failed for package ‘rpud’
* removing ‘/home/ubuntu/R/x86_64-pc-linux-gnu-library/3.1’/rpud’

So I try to find out what -Wl is doing which leads me to another dead-end here http://www.talkstats.com/showthread.php/43438-installing-rpud-got-unrecognized-option-Wl. A friend of mine pointed me to http://gcc.gnu.org/onlinedocs/gcc/Link-Options.html for more information on -Wl, but changing the source-code of rpud or finding/installing the correct linker might be just outside my abilities.

With regard to the comment:

ubuntu@ip-xx-xx-xx:~$ dpkg -l | grep nvidia
ii  nvidia-current                   319.37-0ubuntu1                   NVIDIA binary Xorg driver, kernel module and VDPAU library
ii  nvidia-current-dev               319.37-0ubuntu1                   NVIDIA binary Xorg driver development files
ii  nvidia-modprobe                  319.37-0ubuntu1                   Load the NVIDIA kernel driver and create device files
ii  nvidia-settings                  319.37-0ubuntu1                   Tool for configuring the NVIDIA graphics driver

I am hoping someone is using rpud on ubuntu and could provide any further guidance on how to get rpud working. Thanks in advance for your time. If you need any more information feel free to comment.

EDIT 4/8/2014

Following the python walkthrough, http://enja.org/category/tutorial/advcl/, I am able to get a simple program running on the GPU, the python/part1 example runs perfectly. So I know that the NVIDIA drivers are working properly, at least for python. However I have yet to locate an R walk-through that will even get the packages loaded correctly.

Further findings from Python exploration: I have 2 devices on my current machine that both work.

Choose device(s):
[0] <pyopencl.Device 'Tesla M2050' on 'NVIDIA CUDA' at 0x2806460>
[1] <pyopencl.Device 'Tesla M2050' on 'NVIDIA CUDA' at 0x28064b0>

*Those could be seen by running any of the python scripts from the python GPU tutorial

Edit 4/9/2014

Knowing that Python was interfacing with OpenCL made me think, couldn't R do the same? Obviously someone else had thought the same thing and built the package 'OpenCL'

So I ran install.packages('OpenCL') and IT WORKED. Then running some of the sample code from http://cran.r-project.org/web/packages/OpenCL/OpenCL.pdf ALSO WORKED.

At this point, the only question I'm left with is, has anyone else succeeded with interfacing the GPU with R and if so, how did they do it?

I will post my step-by-step as an answer, but would love to see other ways.

Answer 1

Walkthrough:

Step 1: Lookup the AMI-ID ami-87377cee (the one Erik Hazzard built at http://vasir.net/blog/opencl/installing-cuda-opencl-pyopencl-on-aws-ec2) in AWS in the Community AMIs and start up a cg1.4xlarge machine.

Step 2: From command line run: sudo apt-get update then sudo apt-get install r-base-core

** this will install R2.14.1. If you want to use the latest R version, I would use the guide here: How to install R version 3

Step 3: run R, then use install.packages('OpenCL') to install OpenCL

Step 4: Have fun learning OpenCL!!

It is really that easy to get it working. Writing the code in a way that OpenCL can use is a bit tricky, but once you get the hang of it utilizing the GPU can be a very powerful tool.

See http://cran.r-project.org/web/packages/OpenCL/OpenCL.pdf for some code snippets to get you started.

With this machine you can also easily use Python with OpenCL, where I would recommend: http://enja.org/category/tutorial/advcl/ if you want to go that route.

Answer 2

My solution may apply on your case. I installed successfully by resolving two errors messages. First error message I resolved comes from the source file, rpudist.cu (in src folder), as the error message suggests it is in line 159. You can use a text editor to read the source file and find this code, (dev = 1.).

rpudist.cu(159): warning: use of "=" where "==" may have been intended

So I changed it to (dev == 1.), the error message was then gone.

The second error message, indeed as you have found out, is about -Wl. I think this may be more critical. It seems to conflict with another linker option -Xlinker, which is used in the file, Makefile.in in the src folder of the rpud folder (if you extract the tarball rpud_0.0.2.tar.gz).

LD_PARAMS := -Xlinker "@R_LIB@ @RPATHFLAG@"    

As explained in the gcc doc and I replicate here, both "Pass option as an option to the linker". So I think they passed options after them to ld to link with the files nvcc has compiled. In the following code, nvcc calls both -Xlinker, and -Wl

/usr/local/cuda/bin/nvcc -shared -Xlinker "-Wl,--export-dynamic-fopenmp  -L/usr/lib/R/lib -lR -lpcre -llzma -lbz2 -lrt -ldl -lm -Wl,-rpath,/usr/local/cuda/lib64" -L/usr/local/cuda/lib64 -lcublas -lcuda rpud.o rpudist.o -o rpud.so

Thus, the not-very-elegant workaround is to make nvcc only use -Xlinker. To sum up, except changing the (maybe not critical) file, rpudist.cu, the solution is to alter the contents in the files (1) Makefile.in (in src folder) and (2) configure (in top-level folder).

Changing the line 10 in original Makefile.in from

LD_PARAMS := -Xlinker "@R_LIB@ @RPATHFLAG@"

to:

LD_PARAMS := -Xlinker @R_LIB@ -Xlinker @RPATHFLAG@

Then change the line 1786 in the original configure from,

R_LIB=`"${R_HOME}/bin/R" CMD config --ldflags`

to

R_LIB="-E -fopenmp  -L/usr/lib/R/lib -lR -lpcre -llzma -lbz2 -lz -lrt -ldl -lm"

and line 1797 from,

RPATHFLAG="-Wl,-rpath,${CUDA_HOME}${CUDA_LIB_DIR}"

to

RPATHFLAG="-rpath=${CUDA_HOME}${CUDA_LIB_DIR}"

Finally, just follow Chi Yau's installation instruction

3) Expand the package in a temporary folder:

tar xf rpud_<version>.tar.gz

4) Run configure in rpud:

cd rpud
./configure
cd ..

5) Then enter the following:

R CMD INSTALL rpud

HTH