After some research, tries and errors, I decided to go with UTF8-CPP, which is a lightweight, header-only set of functions for converting to/from utf8. It includes functions for converting from utf-16 to utf-8 and, from my understanding, can deal correctly with BOM.
Then I store all strings as std::string
, converting utf-16 strings to utf-8, something like this (from my example above):
int length; char encoding; char* bytes;
string value;
switch(encoding) {
case 0x00:
case 0x03:
value = string(bytes, length);
break;
case 0x01:
case 0x02:
vector<unsigned char> utf8;
wchar_t* input = (wchar_t*)bytes;
utf16to8(input, input + (length >> 1), back_inserter(utf8));
value = string(utf8.start(), utf8.end());
break;
default:
throw ERROR_INVALID_STRING_ENCODING;
}
This works fine in my quick test. I'll need to do more testing before final judgement.