Special characters coming through as ? in SMPP and Java

Question

I've spent a crazy amount of time trying to get special characters to come through properly in our application. Our provider told us to use "GSM0338, also known as ISO-8859". To me, this means ISO-8895-1, since we want spanish characters.

The flow: (Telling you everything, since I've been playing around with this for a while.)

1. Used notepad++ to create the message files in UTF-8 encoding. (No option to save as ISO-8859-1).

2. Sent each file through a quick Java program which converts and writes new files:

   String text = readTheFile(....);
   
   output = text.getBytes("ISO-8859-1");
   FileOutputStream fos = new FileOutputStream(filesPathWithoutName + "\\converted\\" + filename);
   fos.write(output);
   fos.close();
   

3. SMPP test class in another project reads these files:

   private static String readMessageFile(final String filenameOfFirstMessage) throws IOException {
   
       BufferedReader br = new BufferedReader(new FileReader(filenameOfFirstMessage));
       String message;
   
       try {
           StringBuilder sb = new StringBuilder();
           String line = br.readLine();
   
           while (line != null) {
               sb.append(line);
               sb.append("\n");
               line = br.readLine();
           }
           message = sb.toString();
       } finally {
           br.close();
       }
   
       return message;
   }
   

4. Calls send

   public void send(final String message, final String targetPhone) throws MessageException {
       SmppMessage smppMessage = toSmppMessage(message, targetPhone);
       smppSmsService.sendMessage(smppMessage);
   }
   
   private SmppMessage toSmppMessage(final String message, final String targetPhone) {
       SmppMessage smppMessage = new SmppMessage();
       smppMessage.setMessage(message);
       smppMessage.setRecipientAddress(toGsmAddress(targetPhone));
       smppMessage.setSenderAddress(getSenderGsmAddress());
       smppMessage.setMessageType(SmppMessage.MSG_TYPE_DATA);
       smppMessage.setMessageMode(SmppMessage.MSG_MODE_SAF);
       smppMessage.requestStatusReport(true);
       return smppMessage;
   }
   

Problem: SMSs containing letters ñ í ó are delivered, but with these letters displaying as question marks.

Configuration:

smpp.smsc.charset=ISO-8859-1
smpp.data.coding=0x03

Absolutely any help with this would be GREATLY appreciated. Thank you so much for reading.

Answer 1

Well, your provider is wrong. GSM 03.38 is not ISO-8859-1. They are the same up through "Z" (0x5A), but after that they diverge. For instance, in GSM 03.38, ñ is 0x7D, while in ISO-8859-1, it is 0xF1. Since GSM 03.38 is a 7-bit code, anything above 0x7F is going to come out as a "?". Anything after 0x5A is going to come out as something unexpected.

Since Java doesn't usually come with GSM 03.38 support, you're going to have to decode by hand. It shouldn't be too difficult to do, and the following piece of software might already do most of what you need:

• Java GSM 03.38 SMS Character Set Translator

You might also find this translation table between GSM 03.38 and Unicode useful.