I figured out how to do it using the magic of scripting!
You can take a look at my solution here: http://code.google.com/p/unojoy/source/browse/#hg%2FLeoJoy%2FLeoJoy%20Installer.app%2FContents%2FMacOS
The folder structure it's in makes it show up as an app on OSX, and the two scripts (LeoJoy Installer and LeoJoyInstaller.command) copy the files. There's two scripts there because I wanted to have a console window pop up to show the user progress, and that's the only way I could figure out how to do that. But, if you run this app in the same directory as the Arduino app, it copies files from the installer app into the Arduino.app package, updating the Arduino app so I don't have to distribute a whole separate version of Arduino.