Unfortunately, AFAIK, the only way to limit the results of a query is to use Query#setMaxResults()
and this is only applicable to a Query object, not to a subquery.
I suggest to use a workaround, like selecting only the subcategories with your query. Thanks to lazy loading, when returning the results, JPA shouldn't fetch all products for each result (you can check this with some logging), therefore avoiding an unnecessary db load.
Later, when you need the top 5 products for each subcategory, instead of using
SubCategory sc;
List<Product> list = sc.getProducts();
just run a query for each subcategory:
SubCategory sc;
List<Product> list = subCategoryService.getTopProducts(sc);
This shouldn't have a visible worsening of the performances, if the subcategories shown each time are in a reasonable number (like up to 30, as for the results that can be displayed in a paginated data list).