How about:
*.jar
!libs/**/*.jar
The order is important.
Edit
I used your project structure and have the following output after I did a git add
and git status
$ git stat
On branch master
Initial commit
Changes to be committed:
(use "git rm --cached <file>..." to unstage)
new file: .gitignore
new file: libs/goodFile.jar
new file: libs/someFolder/subFolder/alsoGood.jar
new file: libs/someFolder/subFolder/test/anotherFolder/test.jar
$ cat .gitignore
*.jar
!libs/**/*.jar