NSMutableData to NSString conversion fails if special character present

Question

I'm writing an app to interact with a DirecTV receiver through the http server interface. I'm able to query the device and get the SJON formatted response.

Here is my sample code:

- (NSDictionary *) readDVRData
{
    NSMutableData *rData = [[NSMutableData alloc] initWithContentsOfFile:@"./data"];
    //NSLog(@"rData = %@",rData);
    NSString *rString = [[NSString alloc] initWithData:rData encoding:NSUTF8StringEncoding];
    NSLog(@"rString = %@",rString);
    NSError *error;
    SBJSON *json = [[SBJSON new] autorelease];
    NSDictionary *response = [json objectWithString:rString error:&error];
    //NSLog(@"response = %@",response);
    return response;    
}

For testing purposes, I saved the response to the data file and I'm reading from it in the example above. The conversion from NSMutableData to NSString via:

NSString *rString = [[NSString alloc] initWithData:rData encoding:NSUTF8StringEncoding];

fails if the following character is in place. È as in CrÈme Fraiche. For reference this was in the episodeTitle and description from a South Park recording.

If I change this character to just an upper case "E", the conversion works. Is there a different method I should use to catch something like this? Or is there a simple way to remove characters like this? Please keep in mind I will not know before hand if it's and accent mark or something different that might cause the conversion to fail.

Thanks, Dave

Snippet of data file if that is needed. I would include the entire text, but it pushes the character limit above the 30000 limit.

{
      "groupId": "1 1 381C3",
      "groupTitle": "South Park",
      "isRecording": false,
      "numItems": 10,
      "numNewItems": 10,
      "programs": [
        {
          "callsign": "COMHD",
          "contentId": "1 1 381C3 198",
          "description": "\"CrÈme Fraiche\" Animation, Comedy. (2010)  Trey Parker,  Matt Stone. Randy's obsession with the Food Network forces Sharon to explore a new interest of her own. Series.",
          "duration": 1800,
          "episodeTitle": "CrÈme Fraiche",
          "expiration": "0",
          "isHd": true,
          "isPartial": false,
          "isPpv": false,
          "isRecording": false,
          "isViewed": false,
          "isVod": false,
          "keepUntilFull": true,
          "major": 249,
          "minor": 65535,
          "programId": "6937756",
          "rating": "TV-MA-L",
          "startTime": 1290056400,
          "title": "South Park",
          "uniqueId": "5397375192530140068"
        },
        {
          "callsign": "COMHD",
          "contentId": "1 1 381C3 197",
          "description": "\"Coon vs. Coon & Friends\" Animation, Comedy. (2010)  Trey Parker,  Matt Stone. Coon and Friends find themselves at the mercy of Cartman who now has the dark lord doing his bidding; Kenny wrestles with the curse of his super power. Series.",
          "duration": 1800,
          "episodeTitle": "Coon vs. Coon & Friends",
          "expiration": "0",
          "isHd": true,
          "isPartial": false,
          "isPpv": false,
          "isRecording": false,
          "isViewed": false,
          "isVod": false,
          "keepUntilFull": true,
          "major": 249,
          "minor": 65535,
          "programId": "6918387",
          "rating": "TV-MA-L",
          "startTime": 1289444400,
          "title": "South Park",
          "uniqueId": "8476957599452610419"
        },
        {
          "callsign": "COMHD",
          "contentId": "1 1 381C3 196",
          "description": "\"Mysterion Rises\" Animation, Comedy. (2010)  Trey Parker,  Matt Stone. Mysterion's true identity is revealed; scorned by fellow superheroes, one goes out for revenge. Series.",
          "duration": 1800,
          "episodeTitle": "Mysterion Rises",
          "expiration": "0",
          "isHd": true,
          "isPartial": false,
          "isPpv": false,
          "isRecording": false,
          "isViewed": false,
          "isVod": false,
          "keepUntilFull": true,
          "major": 249,
          "minor": 65535,
          "programId": "6896823",
          "rating": "TV-MA-L",
          "startTime": 1288843200,
          "title": "South Park",
          "uniqueId": "2113731123121326471"
        },
        {
          "callsign": "COMHD",
          "contentId": "1 1 381C3 0",
          "description": "Animation, Comedy.  Trey Parker,  Matt Stone. The adventures of four boys who live in South Park, Colo. Series.",
          "duration": 1800,
          "episodeTitle": "",
          "expiration": "0",
          "isHd": true,
          "isPartial": false,
          "isPpv": false,
          "isRecording": false,
          "isViewed": false,
          "isVod": false,
          "keepUntilFull": true,
          "major": 249,
          "minor": 65535,
          "programId": "6830127",
          "rating": "TV-MA-L",
          "startTime": 1288751400,
          "title": "South Park",
          "uniqueId": "2166056034991125062"
        },
        {
          "callsign": "COMHD",
          "contentId": "1 1 381C3 195",
          "description": "\"Coon 2: Hindsight\" Animation, Comedy. (2010)  Trey Parker,  Matt Stone. Coon and friends set out to help the victims of BP's latest catastrophic drilling accident in the Gulf. Series.",
          "duration": 1800,
          "episodeTitle": "Coon 2: Hindsight",
          "expiration": "0",
          "isHd": true,
          "isPartial": false,
          "isPpv": false,
          "isRecording": false,
          "isViewed": false,
          "isVod": false,
          "keepUntilFull": true,
          "major": 249,
          "minor": 65535,
          "programId": "6876407",
          "rating": "TV-MA-L",
          "startTime": 1288238400,
          "title": "South Park",
          "uniqueId": "8615834721021879366"
        },
        {
          "callsign": "COMHD",
          "contentId": "1 1 381C3 194",
          "description": "\"Insheeption\" Animation, Comedy. (2010)  Trey Parker,  Matt Stone. When Stan is sent to the school counselor because he's holding on to an obscene number of useless possessions, he realizes that Mr. Mackey had a hoarding disorder too. Series.",
          "duration": 1783,
          "episodeTitle": "Insheeption",
          "expiration": "0",
          "isHd": true,
          "isPartial": false,
          "isPpv": false,
          "isRecording": false,
          "isViewed": false,
          "isVod": false,
          "keepUntilFull": true,
          "major": 249,
          "minor": 65535,
          "programId": "6857628",
          "rating": "TV-MA-L",
          "startTime": 1287626416,
          "title": "South Park",
          "uniqueId": "1547798106243799673"
        },
        {
          "callsign": "COMHD",
          "contentId": "1 1 381C3 193",
          "description": "\"It's a Jersey Thing\" Animation, Comedy. (2010)  Trey Parker,  Matt Stone. New Jersey is taking over the nation one state at a time, Randy and the boys take a stand as the Jerseyites approach South Park. Series.",
          "duration": 1800,
          "episodeTitle": "It's a Jersey Thing",
          "expiration": "0",
          "isHd": true,
          "isPartial": false,
          "isPpv": false,
          "isRecording": false,
          "isViewed": false,
          "isVod": false,
          "keepUntilFull": true,
          "major": 249,
          "minor": 65535,
          "programId": "6839243",
          "rating": "TV-MA-L",
          "startTime": 1287028800,
          "title": "South Park",
          "uniqueId": "8123622337314668826"
        },
        {
          "callsign": "COMHD",
          "contentId": "1 1 381C3 193",
          "description": "\"It's a Jersey Thing\" Animation, Comedy. (2010)  Trey Parker,  Matt Stone. New Jersey is taking over the nation one state at a time, Randy and the boys take a stand as the Jerseyites approach South Park. Series.",
          "duration": 1784,
          "episodeTitle": "It's a Jersey Thing",
          "expiration": "0",
          "isHd": true,
          "isPartial": false,
          "isPpv": false,
          "isRecording": false,
          "isViewed": false,
          "isVod": false,
          "keepUntilFull": true,
          "major": 249,
          "minor": 65535,
          "programId": "6839243",
          "rating": "TV-MA-L",
          "startTime": 1287021615,
          "title": "South Park",
          "uniqueId": "949183344652655207"
        },
        {
          "callsign": "COMHD",
          "contentId": "1 1 381C3 0",
          "description": "Animation, Comedy.  Trey Parker,  Matt Stone. The adventures of four boys who live in South Park, Colo. Series.",
          "duration": 1800,
          "episodeTitle": "",
          "expiration": "0",
          "isHd": true,
          "isPartial": false,
          "isPpv": false,
          "isRecording": false,
          "isViewed": false,
          "isVod": false,
          "keepUntilFull": true,
          "major": 249,
          "minor": 65535,
          "programId": "6830127",
          "rating": "TV-MA-L",
          "startTime": 1286937000,
          "title": "South Park",
          "uniqueId": "7841891672059226043"
        },
        {
          "callsign": "COMHD",
          "contentId": "1 1 381C3 0",
          "description": "Animation, Comedy.  Trey Parker,  Matt Stone. The adventures of four boys who live in South Park, Colo. Series.",
          "duration": 1800,
          "episodeTitle": "",
          "expiration": "0",
          "isHd": true,
          "isPartial": false,
          "isPpv": false,
          "isRecording": false,
          "isViewed": false,
          "isVod": false,
          "keepUntilFull": true,
          "major": 249,
          "minor": 65535,
          "programId": "6830127",
          "rating": "TV-MA-L",
          "startTime": 1286856000,
          "title": "South Park",
          "uniqueId": "3872435936942587460"
        }
      ]
    }

Answer 1

Have you tried other encodings, such as NSISOLatin1StringEncoding?

Answer 2

It's likely an encoding problem from the web server. You're expecting UTF-8 data, but I bet you that's not what they are sending you. Try reading using [NSString stringWithContentsOfFile:] instead and see if anything changes.

The compiler will throw a warning because that method is deprecated. In fact, you could use stringWithContentsOfFile:usedEncoding:error: instead, but I and others have reported that it doesn't seem to work as well as the old method for data that is not perfectly encoded (that is, 90% of the web :-)).

Answer 3

Check the HTTP headers in the DirecTV unit's response. It is probably not using UTF-8. Figure out which encoding it is using, then supply the correct constant to initWithData:encoding:.

You can use the command curl -i http://directv-data-url to see the HTTP headers.