Sending a lot of API requests quickly with php?

Question

I'm looking to make a bunch of REST api calls as quickly as possible. I currently have about 1,000 requests to make.

The results of these calls are not needed for any processing or any thing like that. I just simply need to post all of them to the api url.

I of course tried inside my loop which is very slow. I have also tried using curl_multi_exec but that is just about as slow. Here is that code.

foreach($users as $user){
                $ch = curl_init();
                curl_setopt($ch, CURLOPT_USERAGENT, 'Mandrill-PHP/1.0.36');
                curl_setopt($ch, CURLOPT_POST, true);
                curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
                curl_setopt($ch, CURLOPT_HEADER, false);
                curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
                curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
                curl_setopt($ch, CURLOPT_TIMEOUT, 600);
                curl_setopt($ch, CURLOPT_URL, 'https://mandrillapp.com/api/1.0/messages/send.json');
                curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json'));
                curl_setopt($ch, CURLOPT_VERBOSE, TRUE);
                curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
                curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($mandril_message));

                curl_multi_add_handle($mh,$ch);
}
        $active = null;
        //execute the handles
        do {
            $mrc = curl_multi_exec($mh, $active);
        } while ($mrc == CURLM_CALL_MULTI_PERFORM);

        while ($active && $mrc == CURLM_OK) {
            if (curl_multi_select($mh) != -1) {
            do {
                $mrc = curl_multi_exec($mh, $active);
            } while ($mrc == CURLM_CALL_MULTI_PERFORM);
            }
        }

Any thoughts on how I can do this quickly. the rest of my code only takes a second or so to complete without the curl code.

Answer 1

Ok so I finally found some time to get this working correctly using the mandril api. I went from about 2000 api request to 1. Processing time went from 10 minutes to 6.8 seconds.

So basically you just need to make sure you use there merge vars and specify a recipient for each set of merge vars.

Here is an example of how to do this.

"to": [
        {
            "email": "recipient1.email@example.com"
        },
        {
            "email": "recipient2.email@example.com"
        }
    ],
"merge_vars": [
        {
            "rcpt": "recipient1.email@example.com",
            "vars": [
                {
                    "name": "FNAME",
                    "content": "John"
                },
                {
                    "name": "FEED",
                    "content": "Your personalized feed content here"
                }
            ],
            "rcpt": "recipient2.email@example.com",
            "vars": [
                {
                    "name": "FNAME",
                    "content": "Jane"
                },
                {
                    "name": "FEED",
                    "content": "Your personalized feed content here"
                }
            ]
        }
    ]

Answer 2

A thousand REST requests is going to be slow. The trick here is to run the job asynchronously so that the user isn't impacted. You can send an AJAX request to the PHP script in question and enhance the PHP script to write progress data somewhere ($_SESSION, flat file, database, whatever.) You could then send separate AJAX calls to fetch the progress of the job and show updates to the user as the job runs.

This is all assuming that this is a web page we are talking about here and not a shell script.