In the end I solved the problem by myself. The problem was due to the fact that you need to use transaction to use the full speed of the printer. So something like this:
printer.transactionPrint(POSPrinterConst.PTR_S_RECEIPT, POSPrinterConst.PTR_TP_TRANSACTION);
and
printer.transactionPrint(POSPrinterConst.PTR_S_RECEIPT, POSPrinterConst.PTR_TP_NORMAL);
This solves the problem permanently.