UTF 16 to UTF8 using utf8 library in c++

Question

I am using this library to do my conversions from UTF16 to UTF8 in C++.

The example suggests the following way to convert utf16 to utf8:

unsigned short utf16string[] = {0x41, 0x0448, 0x65e5, 0xd834, 0xdd1e};
vector<unsigned char> utf8result;
utf16to8(utf16string, utf16string + 5, back_inserter(utf8result));
assert (utf8result.size() == 10);    

where the definition of utf16to8 is given by:

template <typename u16bit_iterator, typename octet_iterator>
octet_iterator utf16to8 (u16bit_iterator start, u16bit_iterator end, octet_iterator result);

I have a char array which has the characters in UTF16. Can you tell me if it is still possible to use this library if I do not know the size (number of valid characters) of my UTF16 character array ?

Answer 1

No. You cannot do anything meaningful with the data stored in a container of unknown size, obviously. You should know how many elements it holds.

Answer 2

For your reference, probably you can use u16string which is introduced in C++11.

#ifdef   WIN32     
#include <codecvt>
#else
#include <uchar.h>
#endif

string toUTF8(const u16string& u16str) {
    string result;

#ifdef   WIN32  
    wstring_convert<codecvt_utf8_utf16<char16_t>, char16_t> convertor;
    result = convertor.to_bytes(u16str);
#else
    mbstate_t mbs;

    mbrlen(NULL, 0, &mbs);   /* initialize mbs */

    int length = 0;
    char buffer [MB_CUR_MAX];

    for (int i= 0; i < u16str.size(); i++){
        length = c16rtomb(buffer, u16str[i], &mbs);

        if ((length == 0) || (length>MB_CUR_MAX)){
            break;
        }

        for (int j = 0; j < length;j++){
            result += buffer[j];
        }
    }
#endif

    return result;
}