Freepascal, Indy SVN trunk version, nothing received by PHP client when transfering XML, until removing encoding conversion

Question

I have written a server daemon (Linux, Ubuntu) which communicates with PHP as frontend layer.

Recently, i updated both FPC and the Indy library to its FPC 2.6.0 and Indy to the trunk version (before i was using the Tiburon branch).

All compiled, and everything looked fine, but, when writing to an IOHandler, nothing gets received (by the PHP client), the client will report that 0 bytes were received.

After diving into the problem, i saw that when using the write methods from the IOHandler, the encoding is validated and converted before the response is sent, in the ToBytes() method in IdGlobal.pas.

Now if i comment out the conversion lines in the ToBytes() routines;

if ASrcEncoding <> ADestEncoding then begin
  LBytes := TIdTextEncoding.Convert(ASrcEncoding, ADestEncoding, LBytes);

This time, the PHP client receives the response.

My question is, how can i configure my Indy tcp server or IOHandlers to stop encoding the data ?

Answer 1

Indy calls TIdTextEncoding.Convert() when it thinks the two encodings are different so bytes can be converted from one charset to another. However, Indy does not yet detect when two TIdTextEncoding objects represent the same charset so the conversion can be skipped. That is mainly due to a limitation in Embarcadero's SysUtils.TEncoding class in Delphi 2009-XE, which does not expose that information (in Delphi XE2, TEncoding received new EncodingName and CodePage properties, but Indy has not been updated to utilize them yet). Indy's TIdTextEncoding class is an alias for TEncoding in Delphi 2009+, and is modeled after TEncoding in Delphi 5-2007 and FreePascal, in order to maintain a single API throughout Indy's codepage.

Indy currently just compares TIdTextEncoding object pointers to each other, which is fine when using the standard encodings from the TIdTextEncoding class properties, as they are implemented as singleton objects in memory. However, if you mix in TIdTextEncoding objects that are obtained by the TIdTextEncoding.GetEncoding() method, such as from Indy's CharsetToEncoding() function, then the object pointers will not match even if their charsets do. In ideal conditions, that would be a no-op conversion from charset to Unicode back to the same charset.

However, under FreePascal, TIdTextEncoding uses the ICONV library, and Indy's ICONV support is incomplete. Conversions are implemented, but full error handling is not implemented yet, largely due to issues with accessing the errno variable on different platforms, which ICONV uses for extended error reporting. Not all of ICONV's errors are fatal, but Indy cannot detect them yet.

Worse, TEncoding is set up to NOT throw exceptions when conversion errors occur, only when buffer errors occur (shame on Embarcadero for that). If a data conversion error occurs, TEncoding just returns empty data. We had to maintain that behavior in TIdTextEncoding under non-D2009+ environments, like FreePascal. I suppose Indy could be updated to check for that condition internally and raise its own exception when needed.

To answer your question, there is nothing you can do to tell Indy to skip the call to TIdTextEncoding.Convert(). You would have to comment it out and recompile Indy for the time being. This is a known issue in the current Indy release, and there has been some work done to address it, but there is no ETA yet on when that will be ready for public use. In Indy 11, we are likely going to drop support for TEncoding and implement our own charset engine natively in Indy, at least for commonly used charsets. That way we are not tied to any particular platform-specific APIs anymore. But we have not even started work on Indy 11 yet, or even decided what its feature set will be.