The timings you've taken aren't particularly meaningful because the code isn't warm yet. You need to "warm up" the code so that the JVM will JIT-compile it, and that's when you should start seeing good speed. You should look at How to benchmark functions in Clojure? (You should use Criterium.)
As for your code, you're using arrays, so that should give you good performance. Style-wise, the two hanging ]
you have are really weird. Maybe that's just a formatting error? It's usually good to eliminate as much duplicate code as possible, so I'd also change this
(if (pxl-over-threshold? idx)
(aset pxls idx (color 255))
(aset pxls idx (color 0)))
to this
(aset pxls idx (color (if (pxl-over-threshold? idx) 255 0)))
If you feel looks too confusing/complex (I'm kind of right on the edge as to whether I'd consider that too hard to read or not), you could alternatively write it either of these ways instead:
(let [c (if (pxl-over-threshold? idx) 255 0)]
(aset pxls idx (color c)))
(->> (if (pxl-over-threshold? idx) 255 0) color (aset pxls idx))