OK I've found the solution here: https://developers.facebook.com/docs/reference/plugins/like/
Instead of checking for IP or referrer, we can check for the user agent ($_SERVER['HTTP_USER_AGENT']
) of the FB scraper. The value is facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
Hope it helps someone else.