I solved this problem by creating a new Java Library Module, and putting all my 3rd party jars in there. I didn't add any java code of my own in that module. Then I made my Android Library module dependent on the the module. My new project structure is
Project
- AndroidApp
- AndroidLibrary
- build
- src
- (library source code)
- Java Library
- libs
- acra-4.5.0.jar
- activation.jar
- (More jar files)