Read large text file in C on ios

Question

I have a vb6 program that reads data from sql server and writes them to a text file. Each record is delimited by the newline character. These files (can also be> 200mb) must be read from and written to the iPad in a sqlite database. To avoid memory warning I read every single line of the file with this function in C

"strRet" is the string read in C

"NSString *stringa" is the string C transformed into NSString

NSDictionary *readLineAsNSString(FILE *f,int pospass,BOOL testata,int primorecord  )
{
    char *strRet = malloc(BUFSIZ);//(char *) togliere perche con c potrebbe restituire un int
    if (strRet==NULL)
    {
        return nil;
    }

    int size = BUFSIZ;

    BOOL finito=NO;
    int pos = 0;
    int c;
    fseek(f,pospass,SEEK_SET);

    do{ // read one line

        c = fgetc(f);

        if (pos >= size-1)
        {
            size=size+BUFSIZ;
            strRet = realloc(strRet, size);
            if (strRet==NULL)
            {
                return nil;
            }

        }

        if(c != EOF)
        {
            strRet[pos] = c;
            pos=pos+1;
        }
        else
        {
            finito=YES;
        }

    }while(c != EOF && c != '\n');

    if (pos!=0)
    {
        strRet[pos] = '\0';
    }

    NSString *stringa=[NSString stringWithCString:strRet encoding:NSASCIIStringEncoding];

    if (pos==0)
    {
        stringa=@"";
    }

    long long sizerecord;
    if (pos!=0)
    {
        sizerecord=   (long long) [[NSString stringWithFormat:@"%ld",sizeof(char)*(pos)] longLongValue];
    }
    else
    {
        sizerecord=0;
    }
    pos = pospass + pos;

    NSDictionary *risultatoc = @{st_risultatofunzione: stringa,
                                 st_criterio: [NSString stringWithFormat:@"%d",pos],
                                 st_finito: [NSNumber numberWithBool:finito],
                                 st_size: [NSNumber numberWithLongLong: sizerecord]
                                 };

    //Svuoto il buffer
    free(strRet);
    // free(tmpStr);
    strRet=NULL;

    return risultatoc;

}

However, it happens that when I have a special character in the file (such as the € symbol or accented letters or some of the Nordic countries) the record is not read correctly, and I find myself with a NSString of random characters instead of the right one. You know you help me? thank you!

Answer 1

The following line tells iOS that you have ASCII data:

NSString *stringa= [NSString stringWithCString:strRet encoding:NSASCIIStringEncoding];

However, the € symbol or accented letters aren't part of ASCII. So you obviously have a different.

Figure out what encoding it is (e.g. UTF-8, Windows ANSI, ISO-8859-1) and update the line accordingly, e.g.:

NSString *stringa= [NSString stringWithCString:strRet encoding: NSWindowsCP1251StringEncoding];

Update

Figuring out what encoding is being used can be tricky.

From my experience, VB6 and SQL Server are a good pair as they usually don't mess up the encoding. The weak part is the text file, which depends on the encoding but doesn't contain any explicit information about what encoding was used. VB6 probably uses the Windows default setting, which depends on your language setting. Unfortunately, I don't know where you can see the default encoding in Windows.

In Western countries, the encoding is usually set to Windows ANSI aka Code Page 1251 (that's where the constant NSWindowsCP1251StringEncoding comes from).

You can more or less verify it. If you open a text files that contains the Euro symbol (€), it must use the value 80 (hex) if it's encoded in CP 1251. In Latin-1 (aka ISO-8859-1) you cannot represent the euro symbol. In Latin-9 (aka ISO-8859-15), it would use A4 (hex). And in UTF-8, three bytes would be required: E2 82 AC.

So check it yourself. If you're unsure, add a hex dump of a relevant piece of your text file.