iPhone iOS5 CLGeocoder how to geocode a large (200) set of addresses?

Question

I got a large set of about 200 addresses for which I need to know their latitude and longitude. I've created a method that parses the addresses, and now I'm trying to get coordinates for these addresses using CLGeocoder.

My current approach is to create geocoders in parallel and let them do their magic. I noticed that each one of them seems to take a separate thread. (so I saw up to 100 threads at one point).

The problem that I'm running into is that at some point (after about 50 addresses), geocodes stop returning any place marks, and the

NSLog(@"Address not recognized: *%@*",[htc objectForKey:kAddressKey]);

gets called. Is this a limitation on a number of threads or a built-in CLGeocoder limitation? Could it be that I'm not cleaning up geocoders properly and need some sort of an autorelease statement(ARC)?

-(void)geocodeArray:(NSMutableArray*)array {

    NSMutableDictionary* htc = nil;
    objectsToGeocode = array.count;

    NSDictionary *htcDictionary =nil;
     for (int i = 0; i<array.count;i++) {
         htcDictionary = [array objectAtIndex:i];

        //create an updated dictionary that would hold the reverse geocoding location
        htc = [[NSMutableDictionary alloc] initWithDictionary:htcDictionary];
        NSLog(@"geocoding: %@",[htc objectForKey:kAddressKey]);

        CLGeocoder* geoCoder = [[CLGeocoder alloc] init];
        [geoCoder geocodeAddressString:[htc objectForKey:kAddressKey] completionHandler:^(NSArray *placemarks, NSError *error) {

            if(placemarks.count>0)
            {
                NSLog(@"Found placemarks for %@",[htc objectForKey:kAddressKey]);
                CLPlacemark* placemark =  [placemarks objectAtIndex:0];
                MyLocation *annotation = [[MyLocation alloc]
                                          initWithName:[htcDictionary objectForKey:kNameKey]
                                          address:[htcDictionary objectForKey:kAddressKey]
                                          coordinate:placemark.location.coordinate] ;
                annotation.faxNumber = [htc objectForKey:kFaxKey];
                annotation.phoneNumber = [htc objectForKey:kPhoneKey];
                annotation.website = [htc objectForKey:kWebsiteKey];
                annotation.type = [htc objectForKey:kFacilityTypeKey];
                [_mapView addAnnotation:annotation];  


                double placemarkToUserDistance = [self._mapView.userLocation.location distanceFromLocation:placemark.location] ;
                //convert distance to miles
                placemarkToUserDistance =placemarkToUserDistance/ 1000/ kKilometersPerMile;

                [htc setObject:[NSNumber numberWithDouble:placemarkToUserDistance] forKey:kDistanceToUserKey];
                [htc setObject:[NSNumber numberWithDouble:placemark.location.coordinate.latitude] forKey:kLatitudeKey];
                 [htc setObject:[NSNumber numberWithDouble:placemark.location.coordinate.longitude] forKey:kLongitudeKey];
                NSAssert([htc objectForKey:kLatitudeKey]!=nil,@"kLatitudeKey is not saved!");
                NSAssert([htc objectForKey:kLongitudeKey]!=nil,@"kLongitudeKey is not saved!");

            }else {
                NSLog(@"Address not recognized: *%@*",[htc objectForKey:kAddressKey]);
            }


            [self.dataSource addObject:htc];

            if(++geocodingCount >=objectsToGeocode){
                NSLog(@"%@",self.dataSource);
                    [self saveGeocoding];

            }

        } ];


        //        [temp addObject:htcDictionary];
    }

}

To test if this is a threading issue, I created this method, which splits my large dataset into 5 arrays, and attempts to geocode them in chunks. I noticed that the first request passes, as well as a part of a second one. But once the magic number of (~50) is reached, the geocoding stops.

Any ideas of what may be happening? Is this an Apple imposed limit on the number of geocoding operations? Should I increase the delay between requests or try to run the app 5 separate times and piece together the results by hand?

-(void)geocodeDatasource
{

        //I'm trying to build a file with coordinates of addresses and include it with the app
        geocodingCount = 0;
        self.dataSource = [[NSMutableArray alloc] initWithCapacity:self.arrayForGeocodingInitialJSON.count+5];
        haveToEmailInitialResults = YES;


    //attempt to geocode in batches

     float numberOfArrays = 5.0;
    NSMutableArray* array1 = [[NSMutableArray alloc] initWithCapacity:arrayForGeocodingInitialJSON.count/numberOfArrays];
    NSMutableArray* array2 = [[NSMutableArray alloc] initWithCapacity:arrayForGeocodingInitialJSON.count/numberOfArrays];
    NSMutableArray* array3 = [[NSMutableArray alloc] initWithCapacity:arrayForGeocodingInitialJSON.count/numberOfArrays];
    NSMutableArray* array4 = [[NSMutableArray alloc] initWithCapacity:arrayForGeocodingInitialJSON.count/numberOfArrays];
    NSMutableArray* array5 = [[NSMutableArray alloc] initWithCapacity:arrayForGeocodingInitialJSON.count/numberOfArrays];

    for(int i = 0 ;i<arrayForGeocodingInitialJSON.count;i++)
    {
        id object = [arrayForGeocodingInitialJSON objectAtIndex:i];
        if(i<arrayForGeocodingInitialJSON.count*(1/numberOfArrays))
        {
            [array1 addObject:object];
        }else if(i>=arrayForGeocodingInitialJSON.count/numberOfArrays && i<arrayForGeocodingInitialJSON.count*(2/numberOfArrays))
        {
            [array2 addObject:object];
        }else if(i>=arrayForGeocodingInitialJSON.count*(2/numberOfArrays) && i<arrayForGeocodingInitialJSON.count*(3/numberOfArrays))
        {
            [array3 addObject:object];
        }else if(i>=arrayForGeocodingInitialJSON.count*(3/numberOfArrays) && i<arrayForGeocodingInitialJSON.count*(4/numberOfArrays))
        {
            [array4 addObject:object];
        }else if(i>=arrayForGeocodingInitialJSON.count*(4/numberOfArrays) && i<arrayForGeocodingInitialJSON.count)
        {
            [array5 addObject:object];
        }



    }

    //simple delays eliminate the need for extra variables and notifications
        [self geocodeArray:array2];

        [self performSelector:@selector(geocodeArray:) withObject:array1 afterDelay:15];
        [self performSelector:@selector(geocodeArray:) withObject:array3 afterDelay:30];
        [self performSelector:@selector(geocodeArray:) withObject:array4 afterDelay:45];
        [self performSelector:@selector(geocodeArray:) withObject:array5 afterDelay:45];
}

Thank you!

Answer 1

You can't immediately geocode large sets. iOS throttles you. I have seen that iOS limits you to 50 geocodes at a time, with the "time" factor being an unknown.

I've had a similar problem, but as I needed only to present the geocoding data in a sequence to the user that takes time, I queued all my geocodings.

Effectively, I geocode a chunk of 25 - display the results to the user 1 at a time at about an interval of a half second between each. When I have fewer than 4 left to display, I will geocode the next 25. This continues until everything is geocoded (or in my case, indefinitely).

If you need to have everything geocoded at once, you'll need to chain your geocodings together with delays between each chunk and show some sort of busy indicator until you are done. Which could be some time with large sets of geocodings.

Answer 2

The answer is you are violating the TOS for Apple's geocoder.

From the CLGeocoder documentation.

Applications should be conscious of how they use geocoding. Here are some rules of thumb for using this class effectively:

Send at most one geocoding request for any one user action. If the user performs multiple actions that involve geocoding the same location, reuse the results from the initial geocoding request instead of starting individual requests for each action. When you want to update the user’s current location automatically (such as when the user is moving), issue new geocoding requests only when the user has moved a significant distance and after a reasonable amount of time has passed. For example, in a typical situation, you should not send more than one geocoding request per minute. Do not start a geocoding request at a time when the user will not see the results immediately. For example, do not start a request if your application is inactive or in the background.

Apple would be well within their rights to stop providing responses, cut off your app entirely, or even revoke your developer account.

If you need to process such a large number of address quickly, then you should look into getting a service like Factual for Points of Interest or Data Science Toolkit for postal addresses and geographic regions.

---

Update

Here is the API docs for Places API - Resolve, which is Factual's places geocoder.

---

Update 2

Here is the API for Street Address to Coordinates, which is Data Science Toolkit's location geocoder.