I'm an idiot I got it fixed.
I had to change:
<CustomAction Id="SetRollbackData" Property="RestoreDeprecatedFiles" Value="RootDir=[DEPRECATEDFILESROOTDIR];TempDir=[DEPRECATEDTEMPORARYDIR]" Execute="immediate" Return="ignore"/>
To:
<CustomAction Id="SetRollbackData" Property="RestoreDeprecatedFiles" Value="RootDir=[DEPRECATEDFILESROOTDIR];TempDir=[DEPRECATEDTEMPORARYDIR]" />