Encoding.UTF8.GetString doesn't take into account the Preamble/BOM

Question

In .NET, I'm trying to use Encoding.UTF8.GetString method, which takes a byte array and converts it to a string.

It looks like this method ignores the BOM (Byte Order Mark), which might be a part of a legitimate binary representation of a UTF8 string, and takes it as a character.

I know I can use a TextReader to digest the BOM as needed, but I thought that the GetString method should be some kind of a macro that makes our code shorter.

Am I missing something? Is this like so intentionally?

Here's a reproduction code:

static void Main(string[] args)
{
    string s1 = "abc";
    byte[] abcWithBom;
    using (var ms = new MemoryStream())
    using (var sw = new StreamWriter(ms, new UTF8Encoding(true)))
    {
        sw.Write(s1);
        sw.Flush();
        abcWithBom = ms.ToArray();
        Console.WriteLine(FormatArray(abcWithBom)); // ef, bb, bf, 61, 62, 63
    }

    byte[] abcWithoutBom;
    using (var ms = new MemoryStream())
    using (var sw = new StreamWriter(ms, new UTF8Encoding(false)))
    {
        sw.Write(s1);
        sw.Flush();
        abcWithoutBom = ms.ToArray();
        Console.WriteLine(FormatArray(abcWithoutBom)); // 61, 62, 63
    }

    var restore1 = Encoding.UTF8.GetString(abcWithoutBom);
    Console.WriteLine(restore1.Length); // 3
    Console.WriteLine(restore1); // abc

    var restore2 = Encoding.UTF8.GetString(abcWithBom);
    Console.WriteLine(restore2.Length); // 4 (!)
    Console.WriteLine(restore2); // ?abc
}

private static string FormatArray(byte[] bytes1)
{
    return string.Join(", ", from b in bytes1 select b.ToString("x"));
}

Answer 1

It looks like this method ignores the BOM (Byte Order Mark), which might be a part of a legitimate binary representation of a UTF8 string, and takes it as a character.

It doesn't look like it "ignores" it at all - it faithfully converts it to the BOM character. That's what it is, after all.

If you want to make your code ignore the BOM in any string it converts, that's up to you to do... or use StreamReader.

Note that if you either use Encoding.GetBytes followed by Encoding.GetString or use StreamWriter followed by StreamReader, both forms will either produce then swallow or not produce the BOM. It's only when you mix using a StreamWriter (which uses Encoding.GetPreamble) with a direct Encoding.GetString call that you end up with the "extra" character.

Answer 2

Based on the answer by Jon Skeet (thanks!), this is how I just did it:

var memoryStream = new MemoryStream(byteArray);
var s = new StreamReader(memoryStream).ReadToEnd();

Note that this will probably only work reliably if there is a BOM in the byte array you are reading from. If not, you might want to look into another StreamReader constructor overload which takes an Encoding parameter so you can tell it what the byte array contains.

Answer 3

I know I am kind of late to the party but here's the code I am using (feel free to adapt to C#) if you need:

 Public Function Serialize(Of YourXMLClass)(ByVal obj As YourXMLClass,
                                                      Optional ByVal omitXMLDeclaration As Boolean = True,
                                                      Optional ByVal omitXMLNamespace As Boolean = True) As String

        Dim serializer As New XmlSerializer(obj.GetType)
        Using memStream As New MemoryStream()
            Dim settings As New XmlWriterSettings() With {
                    .Encoding = Encoding.UTF8,
                    .Indent = True,
                    .OmitXmlDeclaration = omitXMLDeclaration}

            Using writer As XmlWriter = XmlWriter.Create(memStream, settings)
                Dim xns As New XmlSerializerNamespaces
                If (omitXMLNamespace) Then xns.Add("", "")
                serializer.Serialize(writer, obj, xns)
            End Using

            Return Encoding.UTF8.GetString(memStream.ToArray())
        End Using
    End Function

 Public Function Deserialize(Of YourXMLClass)(ByVal obj As YourXMLClass, ByVal xml As String) As YourXMLClass
        Dim result As YourXMLClass
        Dim serializer As New XmlSerializer(GetType(YourXMLClass))

        Using memStream As New MemoryStream()
            Dim bytes As Byte() = Encoding.UTF8.GetBytes(xml.ToArray)
            memStream.Write(bytes, 0, bytes.Count)
            memStream.Seek(0, SeekOrigin.Begin)

            Using reader As XmlReader = XmlReader.Create(memStream)
                result = DirectCast(serializer.Deserialize(reader), YourXMLClass)
            End Using

        End Using
        Return result
    End Function

Answer 4

for those who do not want to use streams I found a quite simple solution using Linq:

public static string GetStringExcludeBOMPreamble(this Encoding encoding, byte[] bytes)
{
    var preamble = encoding.GetPreamble();
    if (preamble?.Length > 0 && bytes.Length >= preamble.Length && bytes.Take(preamble.Length).SequenceEqual(preamble))
    {
        return encoding.GetString(bytes, preamble.Length, bytes.Length - preamble.Length);
    }
    else
    {
        return encoding.GetString(bytes);
    }
}