Don't be concerned about performance of List.toArray(), its complexity is linear as it will resort to a single loop internally.
As it is implemented with Arrays.copyOf, which eventually comes to System.arraycopy, that is implemented in native code it could be potentially even faster than a java-level loop.