Can I change mangled C++ DLL export symbols? (Binary-compatible but Source-incompatible)

Question

Our C++ project (still) uses the option Treat wchar_t as Built-in: No (/Zc:wchar_t-) from back in the days when it was compiled on VS6.

This results in wchar_t being just a typedef for unsigned short (WORD).

We would like to change this, so that wchar_t is handled as a proper built-in type -- this would greatly ease integration of (modern) libs.

Problem is, we do link to a few DLLs that we cannot recompile that expose their strings as unsigned short* and their headers use wchar_t*. This will result in linker errors when wchar_t is a built-in type, because obviously the export symbols are different.

Changing the headers would require adding a sort of casting layer -- I certainly don't want to add casts to all code that calls into the classes in these headers.

Is it possible to fix the DLLs so that their export symbols would "pretend" to export the built-in wchar_tinstead of WORD? After all, the two types are 100% binary compatible in VC++.

Any other ideas?

Answer 1

It is fine, they are binary compatible.

The only wall you could run into is the name mangling of the exported function. If the DLLs that you cannot change exported these functions using their decorated C++ name then you'll get a fail whale at runtime when the client cannot find the exported name. This will happen when the programmer didn't use extern "C" or didn't use a .def file to rename the exports. Otherwise easy to see with Dumpbin.exe /exports. Fixing that is painful, you'll need to modify the DLL header to change wchar_t to WORD, a macro can do that, and write little adapter functions that take wchar_t and calls the WORD function by casting.

If these are exported C++ classes then you have a bigger problem. You'll need to create a new import library for these DLLs. Start with the output you get from Dumpbin.exe /exports, giving you the original names. And create a .def file from that, using its foo = bar option to rename a symbol. Create the import library with the lib.exe /def option. Create a little test DLL to figure out exactly how you should rename the mangled name.

Answer 2

Thanks to @HansPassant 's answer I was able to pull this off.

Full steps here for reference:

1. Generate export symbols for DLL: dumpbin /EXPORTS ddao35u.dll
2. replace all unsigned-short-pointer types with wchar_t: This means replacing PAG (for unsigned short*) and PBG (for unsigned short const*) with the wchar_t string-mangle PA_W and PB_W

3. Write a new def file with all symbols, where the functions with strings get replaced names. Example would be:

   LIBRARY ddao35u
   EXPORTS
     ??0CdbBSTR@@QAE@PA_W@Z=??0CdbBSTR@@QAE@PAG@Z  @1   ... replaced
                     ^^^^  =                ^^^
     ??0CdbBookmark@@QAE@ABV0@@Z  @2  ... no strings, no replacement needed
   ...
   

4. Generate import library from def file: lib /DEF:ddao35u.def
5. Reference this new ddao35u.lib from you wchar_t-native VS project linker options instead of the original lib file.

I'm going to dump the whole perl script function I used to do this here:

sub fixModuleLib {
  my $basename = shift;
  my $dllpath = shift;
  # my $basename = "ddao35u";
  print "\n\n";
  my $cmdExport = "dumpbin /EXPORTS $dllpath";
  print "Running >> $cmdExport << for export symbols ...\n";
  die "Input file $dllpath not found!" unless -f "$dllpath"; 
  my @exports  = qx/$cmdExport/;

  print "Open $basename.def for writing ...\n";
  open(my $defh, ">", "$basename.def") or die "Unable to open basename.def for writing: $!";
  print $defh "LIBRARY $basename"."\n";
  print $defh "EXPORTS"."\n";

  my @expEntries;
  my $usCount = 0;
  foreach (@exports) {
    #           1    0 00002050 ??0CdbBSTR@@QAE@PAG@Z
    my $ws = '\s+';
    my $hnum = '[ABCDEFabcfed0123456789]+';
    if (m/$ws(\d+)$ws$hnum$ws$hnum$ws(\?\S+)/) {
      my $ord = $1;
      my $mangle = $2;
      my $defline;
      # PAG = unsigned short*
      # PBG = unsigned short const*
      # at least two @ signs before parameter list (at least in our case)
      if ($mangle =~ m/(.+@.*@.*)(P[AB]G)(.*)/) { # case sensitive
        my $pre = $1;
        my $uptr = $2;
        my $post = $3;
        # print "$ord\t$pre>>>$uptr<<<$post\n";
        my $wptr = $uptr;
        $wptr =~ s/G/_W/;
        my $wchartMangle = "$pre$wptr$post";
        $defline = "  $wchartMangle=$mangle  \@$ord";
        print "CHANGED: $defline\n";
        $usCount += 1;                 
      } else {
        # line doesn't contain unsigned short ptr
        $defline = "  $mangle  \@$ord";
      }
      push @expEntries, $defline;
    }
  }
  print "There are ".scalar(@expEntries)." export lines and $usCount of them contain an unsigned-short-pointer.\n";
  die "No entries found!" unless $usCount>0;
  print "Writing entries to $basename.def ...\n";
  foreach my $entry (@expEntries) {
    print $defh $entry."\n"; 
  }
  close($defh);

  my $cmdLib = "lib /DEF:$basename.def";
  print "Generate Export Lib from DEF-File: >> $cmdLib << ...\n";
  system($cmdLib); 
}