I always go for the principle of least surprise
.
When a developer opens your source they will probably be placed at the top of the source. So first comes the important stuff.
As they scroll down your source they should next see the less important stuff.
Finally, at the end they should find the unimportant stuff.
I leave to you the decision as to whether the inner class is important or less important or unimportant.
I would certainly consider inner classes pretty close to unimportant especially if they are private - however, if is some kind of factory class that creates and manipulates objects of that type then they could be considered important.
In your case - you seem to be using an inner class to sub-categorize methods - I would not use an inner class to do that, I would use a nested package such as com.....util.matrixutil
.