How to convert UTF-8 to US-Ascii in Java

Question

We have a system where customers, mainly European enter texts (in UTF-8) that has to be distributed to different systems, most of them accepting UTF-8, but now we must also distribute the texts to a US system which only accepts US-Ascii 7-bit

So now we'll need to translate all European characters to the nearest US-Ascii. Is there any Java libraries to help with this task?

Right now we've just started adding to a translation table, where Å (swedish AA)->A and so on and where we don't find any match for an entered character, we'll log it and replace with a question mark and try and fix that for the next release, but it seems very inefficient and somebody else must have done something similair before.

Answer 1

The uni2ascii program is written in C, but you could probably convert it to Java with little effort. It contains a large table of approximations (implicitly, in the switch-case statements).

Be aware that there are no universally accepted approximations: Germans want you to replace Ä by AE, Finns and Swedes prefer just A. Your example of Å isn't obvious either: Swedes would probably just drop the ring and use A, but Danes and Norwegians might like the historically more correct AA better.

Answer 2

You can do this with the following (from the NFD example in this Core Java Technology Tech Tip):

public static String decompose(String s) {
    return java.text.Normalizer.normalize(s, java.text.Normalizer.Form.NFD).replaceAll("\\p{InCombiningDiacriticalMarks}+","");
}

Answer 3

Instead of creating your own table, you could instead convert the text to normalization form D, where the characters are represented as a base character plus the diacritics (for instance, "á" will be replaced by "a" followed by a combining acute accent). You can then strip everything which is not an ASCII letter.

The tables still exist, but are now the ones from the Unicode standard.

You could also try NFKD instead of NFD, to catch even more cases.

References:

• http://unicode.org/reports/tr15/
• http://www.siao2.com/2005/02/19/376617.aspx
• http://www.siao2.com/2007/05/14/2629747.aspx

Answer 4

In response to the answer given by Joe Liversedge, the referenced Lucene ISOLatin1AccentFilter no longer exists :

It has been replaced by org.apache.lucene.analysis.ASCIIFoldingFilter :

This class converts alphabetic, numeric, and symbolic Unicode characters which are not in the first 127 ASCII characters (the "Basic Latin" Unicode block) into their ASCII equivalents, if one exists. Characters from the following Unicode blocks are converted; however, only those characters with reasonable ASCII alternatives are converted.

FYI -

Answer 5

This is typically useful in search applications. See the corresponding Lucene ISOLatin1AccentFilter implementation. This isn't really designed for plugging into a random local implementation, but does the trick.

Answer 6

There are some built in functions to do this. The main class involved is CharsetEncoder, which is part of the nio package. A simpler way is String.getBytes(Charset) that can be sent to a ByteArrayOutputStream.

Answer 7

new String("½".getBytes("US-ASCII"))

Answer 8

this is what i use:

<?php
function remove_accent($str)  {
#   http://www.php.net/manual/en/function.preg-replace.php#96586
$a = array('À', 'Á', 'Â', 'Ã', 'Ä', 'Å', 'Æ', 'Ç', 'È', 'É', 'Ê', 'Ë', 'Ì', 'Í', 'Î', 'Ï', 'Ð', 'Ñ', 'Ò', 'Ó', 'Ô', 'Õ', 'Ö', 'Ø', 'Ù', 'Ú', 'Û', 'Ü', 'Ý', 'ß', 'à', 'á', 'â', 'ã', 'ä', 'å', 'æ', 'ç', 'è', 'é', 'ê', 'ë', 'ì', 'í', 'î', 'ï', 'ñ', 'ò', 'ó', 'ô', 'õ', 'ö', 'ø', 'ù', 'ú', 'û', 'ü', 'ý', 'ÿ', 'Ā', 'ā', 'Ă', 'ă', 'Ą', 'ą', 'Ć', 'ć', 'Ĉ', 'ĉ', 'Ċ', 'ċ', 'Č', 'č', 'Ď', 'ď', 'Đ', 'đ', 'Ē', 'ē', 'Ĕ', 'ĕ', 'Ė', 'ė', 'Ę', 'ę', 'Ě', 'ě', 'Ĝ', 'ĝ', 'Ğ', 'ğ', 'Ġ', 'ġ', 'Ģ', 'ģ', 'Ĥ', 'ĥ', 'Ħ', 'ħ', 'Ĩ', 'ĩ', 'Ī', 'ī', 'Ĭ', 'ĭ', 'Į', 'į', 'İ', 'ı', 'IJ', 'ij', 'Ĵ', 'ĵ', 'Ķ', 'ķ', 'Ĺ', 'ĺ', 'Ļ', 'ļ', 'Ľ', 'ľ', 'Ŀ', 'ŀ', 'Ł', 'ł', 'Ń', 'ń', 'Ņ', 'ņ', 'Ň', 'ň', 'ʼn', 'Ō', 'ō', 'Ŏ', 'ŏ', 'Ő', 'ő', 'Œ', 'œ', 'Ŕ', 'ŕ', 'Ŗ', 'ŗ', 'Ř', 'ř', 'Ś', 'ś', 'Ŝ', 'ŝ', 'Ş', 'ş', 'Š', 'š', 'Ţ', 'ţ', 'Ť', 'ť', 'Ŧ', 'ŧ', 'Ũ', 'ũ', 'Ū', 'ū', 'Ŭ', 'ŭ', 'Ů', 'ů', 'Ű', 'ű', 'Ų', 'ų', 'Ŵ', 'ŵ', 'Ŷ', 'ŷ', 'Ÿ', 'Ź', 'ź', 'Ż', 'ż', 'Ž', 'ž', 'ſ', 'ƒ', 'Ơ', 'ơ', 'Ư', 'ư', 'Ǎ', 'ǎ', 'Ǐ', 'ǐ', 'Ǒ', 'ǒ', 'Ǔ', 'ǔ', 'Ǖ', 'ǖ', 'Ǘ', 'ǘ', 'Ǚ', 'ǚ', 'Ǜ', 'ǜ', 'Ǻ', 'ǻ', 'Ǽ', 'ǽ', 'Ǿ', 'ǿ'); 
$b = array('A', 'A', 'A', 'A', 'A', 'A', 'AE', 'C', 'E', 'E', 'E', 'E', 'I', 'I', 'I', 'I', 'D', 'N', 'O', 'O', 'O', 'O', 'O', 'O', 'U', 'U', 'U', 'U', 'Y', 's', 'a', 'a', 'a', 'a', 'a', 'a', 'ae', 'c', 'e', 'e', 'e', 'e', 'i', 'i', 'i', 'i', 'n', 'o', 'o', 'o', 'o', 'o', 'o', 'u', 'u', 'u', 'u', 'y', 'y', 'A', 'a', 'A', 'a', 'A', 'a', 'C', 'c', 'C', 'c', 'C', 'c', 'C', 'c', 'D', 'd', 'D', 'd', 'E', 'e', 'E', 'e', 'E', 'e', 'E', 'e', 'E', 'e', 'G', 'g', 'G', 'g', 'G', 'g', 'G', 'g', 'H', 'h', 'H', 'h', 'I', 'i', 'I', 'i', 'I', 'i', 'I', 'i', 'I', 'i', 'IJ', 'ij', 'J', 'j', 'K', 'k', 'L', 'l', 'L', 'l', 'L', 'l', 'L', 'l', 'l', 'l', 'N', 'n', 'N', 'n', 'N', 'n', 'n', 'O', 'o', 'O', 'o', 'O', 'o', 'OE', 'oe', 'R', 'r', 'R', 'r', 'R', 'r', 'S', 's', 'S', 's', 'S', 's', 'S', 's', 'T', 't', 'T', 't', 'T', 't', 'U', 'u', 'U', 'u', 'U', 'u', 'U', 'u', 'U', 'u', 'U', 'u', 'W', 'w', 'Y', 'y', 'Y', 'Z', 'z', 'Z', 'z', 'Z', 'z', 's', 'f', 'O', 'o', 'U', 'u', 'A', 'a', 'I', 'i', 'O', 'o', 'U', 'u', 'U', 'u', 'U', 'u', 'U', 'u', 'U', 'u', 'A', 'a', 'AE', 'ae', 'O', 'o'); 
return str_replace($a, $b, $str); 
}

function SEOify($i){
#   http://php.ca/manual/en/function.preg-replace.php#90316
$o          = $i;
$o          = html_entity_decode($o,ENT_COMPAT,'UTF-8');
$o          = remove_accent(trim($o)); 
$patterns   = array( "([\40])" , "([^a-zA-Z0-9_-])", "(-{2,})" ); 
$replacers  = array("-", "", "-"); 
$o          = preg_replace($patterns, $replacers, $o);
return $o;
}
?>

Answer 9

This is what seems to work:

private synchronized static String utftoasci(String s){
  final StringBuffer sb = new StringBuffer( s.length() * 2 );

  final StringCharacterIterator iterator = new StringCharacterIterator( s );

  char ch = iterator.current();

  while( ch != StringCharacterIterator.DONE ){
   if(Character.getNumericValue(ch)>0){
    sb.append( ch );
   }else{
    boolean f=false;
    if(Character.toString(ch).equals("Ê")){sb.append("E");f=true;}
    if(Character.toString(ch).equals("È")){sb.append("E");f=true;}
    if(Character.toString(ch).equals("ë")){sb.append("e");f=true;}
    if(Character.toString(ch).equals("é")){sb.append("e");f=true;}
    if(Character.toString(ch).equals("è")){sb.append("e");f=true;}
    if(Character.toString(ch).equals("è")){sb.append("e");f=true;}
    if(Character.toString(ch).equals("Â")){sb.append("A");f=true;}
    if(Character.toString(ch).equals("ä")){sb.append("a");f=true;}
    if(Character.toString(ch).equals("ß")){sb.append("ss");f=true;}
    if(Character.toString(ch).equals("Ç")){sb.append("C");f=true;}
    if(Character.toString(ch).equals("Ö")){sb.append("O");f=true;}
    if(Character.toString(ch).equals("º")){sb.append("");f=true;}
    if(Character.toString(ch).equals("Ó")){sb.append("O");f=true;}
    if(Character.toString(ch).equals("ª")){sb.append("");f=true;}
    if(Character.toString(ch).equals("º")){sb.append("");f=true;}
    if(Character.toString(ch).equals("Ñ")){sb.append("N");f=true;}
    if(Character.toString(ch).equals("É")){sb.append("E");f=true;}
    if(Character.toString(ch).equals("Ä")){sb.append("A");f=true;}
    if(Character.toString(ch).equals("Å")){sb.append("A");f=true;}
    if(Character.toString(ch).equals("ä")){sb.append("a");f=true;}
    if(Character.toString(ch).equals("Ü")){sb.append("U");f=true;}
    if(Character.toString(ch).equals("ö")){sb.append("o");f=true;}
    if(Character.toString(ch).equals("ü")){sb.append("u");f=true;}
    if(Character.toString(ch).equals("á")){sb.append("a");f=true;}
    if(Character.toString(ch).equals("Ó")){sb.append("O");f=true;}
    if(Character.toString(ch).equals("É")){sb.append("E");f=true;}
    if(!f){
     sb.append("?");
    }
   }
   ch = iterator.next();
  }
  return sb.toString();
 }