I still don't know WHY it is that way, but the installation works, if you do it via a scheduled task.
Because I thought the only difference is maybe the account installing the software, I first tried to install the test controller via runas command, but I could not pass a password along with the command. So I created a scheduled task for installing the test controller and was able to pass user credentials along with the command. Now my setupController.cmd looks like this:
powershell $command = "set-executionpolicy Unrestricted"
powershell $command = ".\setupController.ps1" -NonInteractive >; out.txt
echo Setup TestController >> %~dp0out.txt
cd E:\approot
echo Setup has been executed! >> %~dp0out.txt
net user vstestagent MyPassword! /add
net localgroup Administrators vstestagent /add
REM Create a task that will run with full network privileges.
net start Schedule
schtasks /CREATE /TN "Install Test Controller Service" /SC ONCE /SD 01/01/2020 /ST 00:00:00 /RL HIGHEST /RU vstestagent /RP MyPassword! /TR "E:\approot\testcontroller.exe /full /q /Log %~dp0install.log" /F
schtasks /RUN /TN "Install Test Controller Service"
timeout /T 120
schtasks /CREATE /TN "Configure Test Controller Service" /SC ONCE /SD 01/01/2020 /ST 00:00:00 /RL HIGHEST /RU vstestagent /RP MyPassword! /TR e:\approot\configcontroller.cmd /F
schtasks /RUN /TN "Configure Test Controller Service"
Hope it helps somebody else also.