BigCommerce PHP API - Pulling large amounts of data

Question

I'm attempting a couple different data pulls using the BigCommerce PHP API.

In one attempt, I need to pull all of my customers and their addresses. In the other, I need to pull all of my orders and the coupon data (if any) associated with them.

The problem I'm having is a combination of the way BigCommerce returns the data, and with the amount of data I am attempting to pull.

When I pull a list of customers the address data is not stored with the results. Instead I have to query a separate JSON file. Example:

https://STORE-ID.mybigcommerce.com/api/v2/customers/2104/addresses.json

According to the quickstarts and the response from their API team, they expect me to simply iterate through each customer/order ID and then make an additional request to pull the address/coupon data for each ID.

Due to the amount of data I have here, this operation results in either script timeouts (30+ seconds), or in PHP running out of memory.

Yes, I know the general solution with PHP is throw more hardware at it, but there has to be a more efficient way to do this than to simply make a ton of single-shot long requests, right?

I'm thinking something in the way of multiple threads or jobs, though I personally am not aware of any such functionality.

Answer 1

As suggested by Chirag B, I ended up using Node.JS and splitting this into multiple async calls.