Solved this with the internal devs. There was some problem with the Chromedriver update script which had been added by (I'm assuming NuGet)
Starting
<PropertyGroup Label="DownloadChromeDriverBuildTask">
and also:
<Target Name="DownloadChromeDriver" Label="DownloadChromeDriverBuildTask">
It was a small script that was installing ChromeDriver if it was not already available. Removing this seems to have resolved it. Hope this provides help to anyone with a similar issue.