There's a reference manual here: http://www.keplerproject.org/luajava/manual.html. The Java documentation is at the bottom.
Here's an example (although it might be a bit out-dated): http://lua-users.org/lists/lua-l/2003-12/msg00286.html
You might want to start with understanding the C interface to Lua. The LuaJava interface is built on that. The basic idea is that you have a stack and you first push the function, followed by arguments, and then you can execute a function in Lua. The results are then pushed on the stack and you can get data back from Lua that way.