I found the solution. Apparently you need to give each virtual machine config spec its own key, even if it is negative. For posterity set
$disk->key = UNIQUE NEGATIVE NUMBER
and then set your config spec
$devSpec = VirtualDeviceConfigSpec->new(
operation => $devSpec->operation,
fileOperation => $devSpec->fileOperation,
device => $disk,
);