Java does not provide a convenient way to list the "files" in a "directory", when that directory is backed by a JAR file on the classpath (see How do I list the files inside a JAR file? for some work-arounds). I believe this is because the general case where a "directory" exists in multiple .jar files and classpath directories is really complicated (the system would have to present a union of entries across multiple sources and deal with overlaps).
Because Java/Android do not have clean support this, neither does Libgdx (searching the classpath is what "internal" Libgdx files map to).
I think the easiest way to work-around this is to build a list of levels into a text file, then open that and use it as a list of file names. So, something like:
// XXX More pseudo code in Java than actual code ... (this is untested)
fileList = Gdx.files.internal("levels.txt");
String files[] = fileList.readString().split("\\n");
for (String filename: files) {
FileHandle fh = Gdx.files.internal("levels/" + filename);
...
}
Ideally you'd set something up that builds this text file when the JAR file is built. (That's a separate SO question I don't know the answer to ...)
As an aside, your work-around that uses ApplicationType.Desktop
is leveraging the fact that in addition to the .jar file, there is a real directory on the classpath that you can open. That directory doesn't exist on the Android device (it actually only exists on the desktop when running under your build environment).