CoreData SQLite storing image data failure

Question

I've been storing images (JPEG) in CoreData (SQLite store) for a while without problem. I've recently had to import some data from an external source and it appeared to be working fine with the new data. I've just come across an instance of data where the following applies: 1. The entities load fine and have the relevant information 2. The entity relationships are restored 3. The entity that contains the NSData property returns nil for that property 4. When I query the backing store for that object it has binary data showing in the equivalent column

When I've looked into the store further it appears that the problem rows in question only have 38 bytes of information stored in the binary data column. Consistently 30% of all of the images that are processed have this 38 bytes of data stored while the remainder have completely accurate and correct data stored.

The import process runs in a background thread. In order to manage the memory requirements of 4.5Gb of image data I run the import in batches of 100-1000 (initially 1000 but have tried 100 to see if any difference). I create an nsmanagedobjectcontext at the beginning of the batch, process the batch then save the context. During the processing I NSLog the length of data being assigned to the NSData property of the objects in question and they are accurate 100% of the time. However when the data is saved and I look in the store 30% have only 38 bytes of data. I have also tried saving the context after every object is created but have the same 30% dud result.

I have a strong feeling there is an element of multi-threading screwing up my import process but I can't find anything online about this. Would love some input from a CoreData expert as to what the hell is going on with my data!!!

Edit: A subset of the code (there is a lot of other stuff going on in the import process but the image import is reasonably linear)

    NSArray *importLines = [[importFileContents componentsSeparatedByString:@"\r\n"] retain];

    int batch = 100;
    int currentBatch = 0;
    int totalLines = importLines.count;

    while(currentBatch*batch<totalLines)
    {
        NSAutoreleasePool *pool = [[NSAutoreleasePool alloc]init];
        // Create a live context
        NSManagedObjectContext *liveContext = [[PRModel sharedInstance] newLiveManagedObjectContext];
        NSMutableArray *errorMessages = [[NSMutableArray alloc]init];
        for(int i=currentBatch*batch; i<(currentBatch+1)*batch && i<totalLines; i++)
        {
            NSString *importLine = [importLines objectAtIndex:i];

            NSArray *importLineData = [importLine componentsSeparatedByString:@"~"];
            if([importLineData count]>0)
            {
                NSString *lineType = [importLineData objectAtIndex:0];
                int lineTypeID = [lineType intValue];
                NSString *errorMessage = nil;
                switch(lineTypeID)
                {
                    // other cases handle other object types and all work fine
                    case 5:
                        errorMessage = [DataImporter processPhoto:importLineData withRefDate:refDate intoContext:liveContext];
                        // try saving every object to see if it makes an effect
                        [Utils saveManagedObjects:liveContext];
                        break;
                    default:
                        break;
                }
                if(errorMessage!=nil)
                {
                    [errorMessages addObject:errorMessage];
                }
            }
        }
        [Utils saveManagedObjects:liveContext];
        [liveContext release];
        // Update the errors/missing areas/missing items files in case we get a memory crash
        [DataImporter writeErrors:errorMessages];
        [errorMessages release];
        currentBatch++;
        NSLog(@"Batch %d complete",currentBatch);
        [pool release];
    }

PRModel newLiveManagedObjectContext

-(NSManagedObjectContext*)newLiveManagedObjectContext
{
    NSPersistentStoreCoordinator *coordinator = [self livePSC];
    NSManagedObjectContext *newContext = nil;
    if (coordinator != nil) {
        newContext = [[NSManagedObjectContext alloc] init];
        [newContext setPersistentStoreCoordinator: coordinator];
    }
    return newContext;
}
- (NSPersistentStoreCoordinator *)livePSC {

    if (livePSC != nil) {
        return livePSC;
    }

    LiveDatabaseUpgradeController *upgrader = [[LiveDatabaseUpgradeController alloc]initWithDelegate:nil];
    @try{
        livePSC = [[NSPersistentStoreCoordinator alloc] initWithManagedObjectModel:self.currentModel];
        NSError *error = nil;
        if (![persistentStoreCoordinator addPersistentStoreWithType:NSSQLiteStoreType configuration:nil URL:[NSURL fileURLWithPath:[self currentModelStorePath]] options:nil error:&error]) 
        {
            NSLog(@"Failed To Open Data Store. Error:%@, %@", error, [error userInfo]);
            @throw [NSException exceptionWithName:@"Upgrade Required!" reason:@"attempted to open your database but failed. To quit press the home button." userInfo:nil];
        }
    }
    @catch(NSException *e){
        [Utils showAlert:[e name] withMessage:[e reason] withDelegate:nil];
    }
    @finally{
        [upgrader release];
    }
    return livePSC;
}

The processPhoto method

+(NSString*)processPhoto:(NSArray*)data withRefDate:(NSDate*)refDate intoContext:(NSManagedObjectContext*)liveContext// withData:(bool)saveData
{
    if(data.count!=10)
         return [NSString stringWithFormat:@"Expected 10 items, found %d",[data count]];
    NSString *partialPath = [data objectAtIndex:8];
    if([partialPath isEqualToString:@"NULL"])
        return nil;
    NSString *filePath = [[[PRModel sharedInstance] applicationDocumentsDirectory] stringByAppendingPathComponent:partialPath];
    if(![[NSFileManager defaultManager] fileExistsAtPath:filePath])
    {
        return [NSString stringWithFormat:@"File not found for %@",filePath];
    }
    // Got everything we need so create a photo
    PhotoEntity *photo = [NSEntityDescription insertNewObjectForEntityForName:@"Photo" inManagedObjectContext:liveContext];
    photo.photoID = [data objectAtIndex:9];
    photo.imageType = @"PHOTO";
    photo.photoDescription = @"";
    UIImage *photoImage = [[UIImage alloc]initWithContentsOfFile:filePath];
    [photo setImage:photoImage inContext:liveContext];
    [photoImage release];
    return nil;
}

The setImage method is where the data is actually set

-(void)setImage:(UIImage *)image inContext:(NSManagedObjectContext *)context
{        
    if(self.photoData==nil)
    {
        // Create a new photo data entity
        self.photoData = [NSEntityDescription insertNewObjectForEntityForName:@"PhotoData" inManagedObjectContext:context];
    }
UIImage *resizedImage = [image copyImageResizedWithLength:MAX_STORE_RESOLUTION interpolationQuality:kCGInterpolationHigh];
    NSData *data = UIImageJPEGRepresentation(resizedImage, 0.6);
    NSLog(@"ImageLength=%d",[data length]);
    self.photoData.imageData = data;
    [resizedImage release];
}

So this code works in 70% of cases but fails in 30%... There's a whole bunch of other stuff going on around this but the code for handling images has been rock solid for at least 6 months in the field with 500 users so it's definitely related to the fact that I'm using it in a mass object background thread process.

Answer 1

Sigh... It turns out that CoreData has decided to store files > 130Kb as external data now. The 38 bytes I am seeing in the store is the Guid relating to the external file. Now I need to work out why valid data is returning as nil in the main body of my app when there is clearly data present.

Edit: Ok, worked out the missing data issue as well. I need to run the import on the simulator to maximise the available RAM. I then transferred the database (minus external data folder because I didn't realise it was there) to an iPad for testing... When the app loaded the data it attempted to retrieve the external data file but couldn't find it and so loaded the data property as nil. There weren't any exceptions or errors indicating this was the case which is frustrating to say the least.