You have this weird expression in your query:
WHERE product_name, customtitle RLIKE '".$searchstring."'
Try combining them using concat()
first:
(SELECT p.virtuemart_product_id, l.product_name
from #__virtuemart_products p join
#__virtuemart_products_".VMLANG." l
on p.virtuemart_product_id = l.virtuemart_product_id
WHERE concat(product_name, customtitle) RLIKE '".$searchstring."' AND
p.published = '1'
LIMIT 0,".$prods."
)
union
(select p.virtuemart_product_id, l.product_name
from #__virtuemart_products p join
#__virtuemart_products_".VMLANG." l
on p.virtuemart_product_id = l.virtuemart_product_id
where product_sku RLIKE '".$searchstring."' and
p.published = '1'
LIMIT 0,".$prods.
)
You could also compare each one separately:
WHERE (product_name RLIKE '".$searchstring."' OR
customtitle RLIKE '".$searchstring."'
) AND . . .
Note that I also fixed the join
syntax to use explicit joins.