What about using BeforeInstall and AfterInstall parameters for file?
Usage is:
[Files]
Source: "MYDLL.DLL"; DestDir: "{app}"; BeforeInstall: MyBeforeInstall; AfterInstall: MyAfterInstall;
BeforeInstall and AfterInstall functions must not have a return value!
procedure MyBeforeInstall();
begin
// Your code here: If file (old) file exists call UnregisterServer() on old file
// Use function FileExists(const Name: String): Boolean; or similar for it
// Also you can delete the file entirely with function DeleteFile(const FileName: string): Boolean;
// Hint: You can use 'CurrentFileName' variable to get currently processed file
end;
procedure MyAfterInstall();
begin
// Your (new) file was processed and now you can do additional tweaks on it
// 'CurrentFileName' variable is still available
// Setup registers all files with the 'regserver' or 'regtypelib' flags as the last step of installation so in this function the file is still not registered!
end;