http://sourceforge.net/projects/luaj/
Here's a readme for you to go through.
Basically what you want at certain points in your code invoke Lua scripts that can perform AI calculations and spit out the result back to your Java engine. Of course, this all depends on your engine, how flexible it is in terms of possible integration, etc.
EDIT:
You need to give those scripts some context, that is setup Lua environment so that it's aware of your engine (objects/tables, functions). For example, if you want to enable users to move AI character, then you have to register (for instance) an AIEnemy
object in Lua, give it some methods (move
, shoot
), etc. These methods then link (call) to equivalent methods in your Java engine and pass appropriate parameters.
Embedding Lua and making your app work with it is a real fun, though I've never done it in Java (C/C++).
Having said that, it requires some serious thought into design of the integration. Experiment with really simple stuff to get the feel of it, best even outside your game development. Once you get it, move forward to more serious integration.
Just as a sidenote. Don't let Lua simplicity fool you. It is a very powerful language.