Converting special charactes such as ü and à back to their original, latin alphbet counterparts in C#

Question

I have been given an export from a MySQL database that seems to have had it's encoding muddled somewhat over time and contains a mix of HTML char codes such as & uuml; and more problematic characters representing the same letters such as ü and Ã. It is my task to to bring some consistency back to the file and get everything into the correct Latin characters, e.g. ú and ó.

An example of the sort of string I am dealing with is

Desinfektionslösungstücher für Flächen

Which should equate to

50 Tattoo Desinfektionsl ö    sungst ü    cher f ü    r Fl ä    chen 
50 Tattoo Desinfektionsl ö sungst ü cher f ü r Fl ä chen

Is there a method available in C#/.Net 4.5 that would successfully re-encode the likes of ü and Ã to UTF-8?

Else what approach would be advisable?

Also is the paragraph character ¶ in the above example string an actual paragraph character or part of some other character combination?

I have created a lookup table in the case of needing to do find and replace which is below, however I am unsure as to how complete it is.

É -> É
“ -> "
†-> "
Ç -> Ç
à -> Ã
é, 'é
à -> ú -> ú
• -> -
Ø -> Ø
õ -> õ
í -> í
â -> â
ã -> ã
ê -> ê
á -> á
é -> é
ó -> ó
– -> –
ç -> ç
ª -> ª
º -> º
à  -> à

Answer 1

Well, first of all, as the data has been decoded using the wrong encoding, it's likely that some of the characters are impossible to recover. It looks like it's UTF-8 data that incorrectly decoded using an 8-bit encoding.

There is no built in method to recover data like this, because it's not something that you normally do. There is no reliable way to decode the data, because it's already broken.

What you can try, is to encode the data, and decode it using the wrong encoding again, just the other way around:

byte[] data = Encoding.Default.GetBytes(input);
string output = Encoding.UTF8.GetString(data);

The Encoding.Default uses the current ANSI encoding for your system. You can try some different encodings there and see which one gives the best result.

Answer 2

The data is only partly unrecoverable due to Windows-1252 encoding having 5 unassigned slots. Some modifications of Windows-1252 fill these with control characters but those don't make it to posts in Stackoverflow. If modified Windows-1252 has been used you can fully recover as long as you don't lose the hidden control characters in copy pastes.

There is also the non-breaking space character that is ignored or turned into a space usually with copypastes, but that's not an issue when you deal with bytes directly.

The misencoding abuse this string has gone through is:

UTF-8 -> Windows-1252 -> UTF-8 -> Windows-1252

To recover, here is an example:

String a = "Desinfektionslösungstücher für Flächen";
Encoding utf8 = Encoding.GetEncoding(65001);
Encoding win1252 = Encoding.GetEncoding(1252);

string result = utf8.GetString(win1252.GetBytes(utf8.GetString(win1252.GetBytes(a))));

Console.WriteLine(result);
//Desinfektionslösungstücher für Flächen

Answer 3

It's probably windows-1252 encoded string which you read as UTF-8.

As Guffa mentioned data has been corrupted.

Lets take a look on bytes:
ö -> C3B6 in UTF8

in windows-1252 C3 ->Ã B6 ->¶

so ö ->ö

what about all these "ƒÂ":

ƒ ->83 Â ->C2

Honesty i don't know why they appear, but you can try erase them and do some conversions as Guffa mentioned. Good luck

Answer 4

Here you can find a completer list:

http://bueltge.de/wp-content/download/wk/utf-8_kodierungen.pdf

Answer 5

I've been troubled by this char problem before. Solution:

My .(cs)html file was UTF-8; I converted to UTF-8Y (UTF-8 with a BOM).