Ultimately @SebastianH is correct, it doesn't seem possible through the FileFilter. instead I extended the JFileChooser.
javax.swing.JFileChooser fileChooser = new javax.swing.JFileChooser(defaultLocation){
@Override
public boolean isDirectorySelectionEnabled() {
setOpenButtonState(this, false);
File file = getSelectedFile();
if(file == null){
return false;
}
if(file.isDirectory()){
File[] files = file.listFiles();
for(File f : files){
if(f.isFile() && f.getName().equals("smart.workspace")){
setOpenButtonState(this, true);
return true;
}
}
}
return false;
}
private void setOpenButtonState(java.awt.Container c, boolean flag) {
int len = c.getComponentCount();
for (int i = 0; i < len; i++) {
java.awt.Component comp = c.getComponent(i);
if (comp instanceof javax.swing.JButton) {
javax.swing.JButton b = (javax.swing.JButton)comp;
if ( b != null && b.getText() != null && b.getText().equals("Open") ) {
b.setEnabled(flag);
}
} else if (comp instanceof java.awt.Container) {
setOpenButtonState((java.awt.Container) comp, flag);
}
}
}
};
In addition to preventing the selection I added the nicety of disabling the Open button to avoid confusion.
Credit for the disabling open button --> Disable Open button in JFileChooser?