How to handle UTF-8 and ANSI conversion before Delphi 2009?

Question

In Delphi 2009 we have:

RichEdit1.Lines.LoadFromFile(OpenDialog1.FileName,TEncoding.UTF8);
RichEdit1.Lines.SaveToFile(OpenDialog2.FileName,TEncoding.Unicode);    

How do I do that on Delphi 2006 if I do not have TEconding yet?

Is there someway to transport that newer library back there? or is there a solution hidden in the Web?

Answer 1

I believe that UTF8Encode and UTF8Decode were present even before Delphi 2009. So you can decode/encode byte strings manually. (I have done that myself.)

Answer 2

Here is a snippet from a Delphi 7 project of mine:

function LoadFile(const fn: string): WideString;
var
  f:TFileStream;
  src:AnsiString;
  wx:word;
  i,j:integer;
begin
  if FileExists(fn) then
   begin
    f:=TFileStream.Create(fn,fmOpenRead or fmShareDenyNone);
    try
      f.Read(wx,2);
      if wx=$FEFF then
       begin
        //UTF16
        i:=(f.Size div 2)-1;
        SetLength(Result,i);
        f.Read(Result[1],i*2);
        //detect NULL's
        for j:=1 to i do if Result[j]=#0 then Result[j]:=' ';//?
       end
      else
       begin
        i:=0;
        if wx=$BBEF then f.Read(i,1);
        if (wx=$BBEF) and (i=$BF) then
         begin
          //UTF-8
          i:=f.Size-3;
          SetLength(src,i);
          f.Read(src[1],i);
          //detect NULL's
          for j:=1 to i do if src[j]=#0 then src[j]:=' ';//?
          Result:=UTF8Decode(src);
         end
        else
         begin
          //assume current encoding
          f.Position:=0;
          i:=f.Size;
          SetLength(src,i);
          f.Read(src[1],i);
          //detect NULL's
          for j:=1 to i do if src[j]=#0 then src[j]:=' ';//?
          Result:=src;
         end;
       end;
    finally
      f.Free;
    end;
   end
  else
    Result:='';
end;