The size of the "swatches" is controlled by two UI properties, so you can adjust it as needed (default value being 10):
int length = 20;
UIManager.put("ColorChooser.swatchesRecentSwatchSize", new Dimension(length, length));
UIManager.put("ColorChooser.swatchesSwatchSize", new Dimension(length, length));
Customizing the palette itself requires a custom subclass of AbstractColorChooserPanel, mainly a c&p job (because it's package private) of DefaultSwatchChooserPanel in javx.swing.colorchooser. Then replace the default with your custom panel, something along the lines
JColorChooser chooser = new JColorChooser();
List<AbstractColorChooserPanel> choosers =
new ArrayList<>(Arrays.asList(chooser.getChooserPanels()));
choosers.remove(0);
MySwatchChooserPanel swatch = new MySwatchChooserPanel();
choosers.add(0, swatch);
chooser.setChooserPanels(choosers.toArray(new AbstractColorChooserPanel[0]));