Clojure API should feels more natural as it is created with Clojure idioms in mind. Of course, you can still use the Java API but then expect a lot of Java interop functions calls.
Which of course there nothing wrong with that but it is only not a fluent Clojure API.
I don't see performance penalties, JVM is slow to boot no matter what you are using, whether it is Java, Scala, Clojure or JRuby. Clojure is really performant. By the way, do you know that in Clojure you can compile your project to Bytecode format?
Is Clojure as good choice for file downloader project?
I would say definitely!
One main advantage is how Clojure deal with concurrency. If you think about it, your project will do a lot of threading, locking and synchronization (you are building a downloader that can download many files simultaneously, correct)?
In Clojure you will use a higher abstractions such as agents (really convenient for your project), refs and atoms.
I'm not sure about the resources you read about JVM performance and memory management. JVM is a sophisticated piece of software. JVM offers many strategies to manage memory. Some are suitable for desktop applications, others are suitable for servers. You can pick the suitable strategy depending on your application/system requirements.
By the way, are planning to build your application with Swing? If yes and you decided to go with Clojure, then have a look at Seesaw.
Seesaw is a library/DSL for constructing user interfaces in Clojure. It happens to be built on Swing, but please don't hold that against it.