Can I control the mouse cursor from within R?

Question

Is it possible, to control the mouse pointer from the R console?

I have something like this in mind:

move_mouse(x_pos=100,y_pos=200)   # move the mouse pointer to position (100,200)
mouse_left_button_down            # simulate a press of the left button
move_mouse(x_pos=120,y_pos=250)   # move mouse to select something
mouse_release_left_button          # release the pressed button

In MATLAB, something like this is possible with the following code

import java.awt.Robot;
mouse = Robot;
mouse.mouseMove(0, 0);
mouse.mouseMove(100, 200);

I tried a direct conversion of the above into R that looks like this:

install.packages("rJava")          # install package
library(rJava)                     # load package
.jinit()                           # this starts the JVM
jRobot <- .jnew("java/awt/Robot")  # Create object of the Robot class

Once I got jRobot in R, I tried to call its metho "MouseMove(100,200)" using the two command below which both resulted in an error.

jRobot$mouseMove(10,10)

Error in .jcall("RJavaTools", "Ljava/lang/Object;", "invokeMethod", cl,  : 
              java.lang.NoSuchMethodException: No suitable method for the given parameters

or

.jcall(jRobot,, "mouseMove",10,10)
Error in .jcall(jRobot, , "mouseMove", 10, 10) : 
              method mouseMove with signature (DD)V not found

Answer 1

Finally I found the problem. You have to tell R that 100 is an integer, in order to pass it to java correctly.

install.packages("rJava")          # install package
library(rJava)                     # load package
.jinit()                           # this starts the JVM
jRobot <- .jnew("java/awt/Robot")  # Create object of the Robot class

# Let java sleep 500 millis between the simulated mouse events
.jcall(jRobot,, "setAutoDelay",as.integer(500))

# move mouse to 100,200 and select the text up to (100,300)         
.jcall(jRobot,, "mouseMove",as.integer(100),as.integer(200))
.jcall(jRobot,, "mousePress",as.integer(16))
.jcall(jRobot,, "mouseMove",as.integer(100),as.integer(300))
.jcall(jRobot,, "mouseRelease",as.integer(16))

Answer 2

As of 2017, CRAN has a package called rMouse to handle mouse movement.

library(rMouse)
move(0,0)       # move to top left corner (0,0)
move(50,30)     # move to pixel x = 50, y = 30
left()          # left click
right()         # right click

Under the hood it still uses Java's robot.

Similarly, KeyboardSimulator package published in 2018 seems to be doing pretty much the same thing

Answer 3

If you are using Windows, library(KeyboardSimulator) is the easiest way.

You can use it to make a bot in R

see the example: Move Mouse in R (RStudio) make a bot in R (Rstudio) for social media

Answer 4

What operating system? In Linux, you could use xdotool and call it from the R system function.

> mousemove=function(x,y){system(paste0("xdotool mousemove ",x," ",y))}
> mousemove(0,0)
> mousemove(500,500)

Note these are screen coordinates, nothing to do with the coordinates in your R graphics window, but you weren't clear about what you wanted. You might be able to get the screen coords of your R graphics window with some other X11 utilities and position on a plot if that's what you want.

In Windows, there's probably some other mouse-tweaking program you can leech onto. IDK.

xdotool info: http://tuxradar.com/content/xdotool-script-your-mouse

further reading of that article shows how to activate particular windows and do mouse actions in them.