Question

I have a fantasy sports website where I web scraping from the official sports website.

The PHP file works perfectly and extracts data from the official website, but for some reason the UPDATING of the database takes 5 minutes.

There is no delay function or sleep function in the script and I have asked 2 programmers and they don't seem to know the answer.

The PHP scripts is via a query function, ie script.php?url=officialwebsite.com

I've tried the following refresh function:

  1. <meta http-equiv="refresh" content="15">
  2. <meta http-equiv=refresh content="15; url=<?php echo parse_str($_SERVER['QUERY_STRING']); ?>

I have even contacted the webhosting company and they don't believe there is any delay on the Updating of the database.

Strangely, even when I do a MANUAL REFRESH (F5) every 15 secs, it doesn't refresh until after a few minutes. Really strange considering the script works.

The gist of the script include:

error_reporting(0);
set_time_limit(0);

require_once 'inc/simple_html_dom.php';
require_once 'inc/central.config.php';

$url = $_GET['url'];
$indicator = basename($url);
$parseUrl = parse_url($url);
$path = $parseUrl['path']; $delimiter = '/';
$rounds_from_url = explode($delimiter, $path);
$s = sizeof($rounds_from_url);
$urlRound = $rounds_from_url[$s-2];
$contents = getHTMLCurl($url);
$html = str_get_html($contents);

Other Coding

function getHTMLCurl($url){
    $userAgent = 'Googlebot/2.1 (http://www.googlebot.com/bot.html)';
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
    curl_setopt($ch, CURLOPT_URL,$url);
    curl_setopt($ch, CURLOPT_FAILONERROR, true);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
    curl_setopt($ch, CURLOPT_AUTOREFERER, true);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
    return curl_exec($ch);
}   

THANK YOU IN ADVANCE for reading this.

I have no idea why the script can run perfectly, but just a delay with updating.

No correct solution

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top