Using Json to get data from results:next_page

Question

I currently have an issue with limitation where I get 1000 results per Curl request, the issue here is that I need to get about 7 thousand results.

The data retrieved from the curl is in Json format and at the bottom of the data there is a link to retrieve the next page of data, each page contains 1000 results.

Is it possible to do an array or some sort of loop where I get all data from each page until there is no more next_page URL at the bottom of the last page.

Sample of the code I'm trying to use is:

function curlWrap($url, $json, $action)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_MAXREDIRS, 10 );
curl_setopt($ch, CURLOPT_URL, ZDURL.$url);
curl_setopt($ch, CURLOPT_USERPWD, ZDUSER."/token:".ZDAPIKEY);
switch($action){
case "POST":
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_POSTFIELDS, $json);
break;
case "GET":
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "GET");
break;
case "PUT":
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "PUT");
curl_setopt($ch, CURLOPT_POSTFIELDS, $json);
default:
break;
}
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-type: application/json'));
curl_setopt($ch, CURLOPT_USERAGENT, "MozillaXYZ/1.0");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 20);
$output = curl_exec($ch);
curl_close($ch);
$decoded = json_decode($output);
return $decoded;
}

I just want to find a way of adding an array or loop to keep getting the results/data from the next_page url until there is no more next_page url at the bottom of the page.

A little more detail about the json layout:

The json key is "next_page" & "end_time":

"end_time": ("unix-timestamp") 
"next_page": "https:url/"unix-timestamp" 

After each page there is a new timestamp which will be listed in the key "end_time".

This will then be listed in the end of the "next_page" url. Both of these keys are the last 2 lines in the json code, there is about 20'000 lines of data in the json file altogether.

If any more detail is needed please let me know, thanks in advance.

Answer 1

Without knowing the json layout. how deep the url is, what the key is etc

something like.

function getNext($jsonIn){
    if (!empty($jsonIn['url'])) {
        curlWrap($jsonIn['url'], $json, $action);
    }  
}

//-------------

getNext($decoded);